[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490327694 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003137] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008354] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.022848] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026424] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] *** VALIDATE tmpfs *** [ 0.030495] *** VALIDATE proc *** [ 0.032214] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041165] debug: unmapping init [mem 0xffffffffb7859000-0xffffffffb7860fff] [ 0.043874] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044653] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049012] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052299] rcu: Hierarchical SRCU implementation. [ 0.054392] smp: Bringing up secondary CPUs ... [ 0.055437] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059520] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150901] node 0 deferred pages initialised in 86ms [ 0.154112] devtmpfs: initialized [ 0.155262] x86/mm: Memory block size: 128MB [ 0.158898] gcov: version magic: 0x41383552 [ 0.161293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163305] pinctrl core: initialized pinctrl subsystem [ 0.164162] [ 0.164743] ************************************************************* [ 0.165016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166013] ** ** [ 0.167014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168014] ** ** [ 0.169014] ** This means that this kernel is built to expose internal ** [ 0.170014] ** IOMMU data structures, which may compromise security on ** [ 0.171013] ** your system. ** [ 0.172011] ** ** [ 0.173014] ** If you see this message and you are not debugging the ** [ 0.174016] ** kernel, report this immediately to your vendor! ** [ 0.175016] ** ** [ 0.176013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177019] ************************************************************* [ 0.178832] NET: Registered protocol family 16 [ 0.179471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182459] cpuidle: using governor menu [ 0.184629] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187630] PCI: Using configuration type 1 for base access [ 0.190137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206036] cryptd: max_cpu_qlen set to 1000 [ 0.208161] ACPI: Added _OSI(Module Device) [ 0.209017] ACPI: Added _OSI(Processor Device) [ 0.210011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217463] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223539] ACPI: Interpreter enabled [ 0.225052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226011] ACPI: Using IOAPIC for interrupt routing [ 0.228093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241491] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255337] acpiphp: Slot [2] registered [ 0.256122] acpiphp: Slot [5] registered [ 0.258125] acpiphp: Slot [6] registered [ 0.259113] acpiphp: Slot [7] registered [ 0.261137] acpiphp: Slot [8] registered [ 0.262094] acpiphp: Slot [9] registered [ 0.263107] acpiphp: Slot [10] registered [ 0.265155] acpiphp: Slot [3] registered [ 0.266105] acpiphp: Slot [4] registered [ 0.268101] acpiphp: Slot [11] registered [ 0.269102] acpiphp: Slot [12] registered [ 0.271122] acpiphp: Slot [13] registered [ 0.272106] acpiphp: Slot [14] registered [ 0.274124] acpiphp: Slot [15] registered [ 0.275097] acpiphp: Slot [16] registered [ 0.276097] acpiphp: Slot [17] registered [ 0.278129] acpiphp: Slot [18] registered [ 0.279091] acpiphp: Slot [19] registered [ 0.281099] acpiphp: Slot [20] registered [ 0.282103] acpiphp: Slot [21] registered [ 0.283138] acpiphp: Slot [22] registered [ 0.285075] acpiphp: Slot [23] registered [ 0.286127] acpiphp: Slot [24] registered [ 0.287075] acpiphp: Slot [25] registered [ 0.289073] acpiphp: Slot [26] registered [ 0.290089] acpiphp: Slot [27] registered [ 0.291083] acpiphp: Slot [28] registered [ 0.292086] acpiphp: Slot [29] registered [ 0.293110] acpiphp: Slot [30] registered [ 0.295088] acpiphp: Slot [31] registered [ 0.296082] PCI host bridge to bus 0000:00 [ 0.297021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311964] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314316] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328511] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.337361] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339625] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363900] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368932] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374278] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.405015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414615] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.420016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456933] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.462028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.468021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.491265] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.519018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.529728] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.550022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.565017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.577457] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.582015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.589018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.602017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.612089] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.615422] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.617367] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.620265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.621159] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.625113] iommu: Default domain type: Passthrough [ 0.626290] SCSI subsystem initialized [ 0.627079] ACPI: bus type USB registered [ 0.628026] usbcore: registered new interface driver usbfs [ 0.629069] usbcore: registered new interface driver hub [ 0.631076] usbcore: registered new device driver usb [ 0.632130] pps_core: LinuxPPS API ver. 1 registered [ 0.633007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.635057] PTP clock support registered [ 0.637068] EDAC MC: Ver: 3.0.0 [ 0.638351] PCI: Using ACPI for IRQ routing [ 0.640203] NetLabel: Initializing [ 0.641010] NetLabel: domain hash size = 128 [ 0.642010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.643153] NetLabel: unlabeled traffic allowed by default [ 0.646040] vgaarb: loaded [ 0.647189] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.648009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.651306] clocksource: Switched to clocksource kvm-clock [ 0.765165] VFS: Disk quotas dquot_6.6.0 [ 0.766643] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.768961] *** VALIDATE ramfs *** [ 0.770041] *** VALIDATE hugetlbfs *** [ 0.771589] pnp: PnP ACPI init [ 0.773928] pnp: PnP ACPI: found 6 devices [ 0.801520] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.804826] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.807328] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.809751] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.812194] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.815062] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.818225] NET: Registered protocol family 2 [ 0.820903] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.825982] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.831381] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.836966] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.840497] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843083] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.846283] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.848941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851979] NET: Registered protocol family 1 [ 0.854462] RPC: Registered named UNIX socket transport module. [ 0.856523] RPC: Registered udp transport module. [ 0.858299] RPC: Registered tcp transport module. [ 0.860062] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.862620] NET: Registered protocol family 44 [ 0.863651] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.867481] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.869622] PCI: CLS 0 bytes, default 64 [ 0.871161] Unpacking initramfs... [ 2.300987] debug: unmapping init [mem 0xffff9c1a3cc54000-0xffff9c1a3ffbffff] [ 2.307401] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.309995] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.314529] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.822369] Initialise system trusted keyrings [ 2.823983] Key type blacklist registered [ 2.826991] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.839160] zbud: loaded [ 2.842739] *** VALIDATE nfs *** [ 2.843850] *** VALIDATE nfs4 *** [ 2.845457] pstore: using deflate compression [ 2.849369] Platform Keyring initialized [ 2.952334] NET: Registered protocol family 38 [ 2.954414] Key type asymmetric registered [ 2.955890] Asymmetric key parser 'x509' registered [ 2.957607] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.961417] io scheduler mq-deadline registered [ 2.963609] io scheduler kyber registered [ 2.965403] io scheduler bfq registered [ 2.967288] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970585] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973553] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.976404] ACPI: Power Button [PWRF] [ 2.982156] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.989398] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.001629] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.009483] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.027332] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.053863] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.080885] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.110492] Non-volatile memory driver v1.3 [ 3.112884] Linux agpgart interface v0.103 [ 3.148055] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.151819] vda: detected capacity change from 0 to 76521472 [ 3.164894] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.167912] vdb: detected capacity change from 0 to 1073741824 [ 3.181505] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.185133] vdc: detected capacity change from 0 to 2621440000 [ 3.199666] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.202024] vdd: detected capacity change from 0 to 2621440000 [ 3.217467] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.219816] vde: detected capacity change from 0 to 4294967296 [ 3.235760] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238119] vdf: detected capacity change from 0 to 4294967296 [ 3.245530] libphy: Fixed MDIO Bus: probed [ 3.254445] usbcore: registered new interface driver usbserial_generic [ 3.256970] usbserial: USB Serial support registered for generic [ 3.259210] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.263462] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.265421] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.268325] mousedev: PS/2 mouse device common for all mice [ 3.271165] rtc_cmos 00:05: RTC can wake from S4 [ 3.273273] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.275083] rtc_cmos 00:05: registered as rtc0 [ 3.280203] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.280701] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.286184] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.287240] intel_pstate: CPU model not supported [ 3.293129] hid: raw HID events driver (C) Jiri Kosina [ 3.295066] usbcore: registered new interface driver usbhid [ 3.297012] usbhid: USB HID core driver [ 3.298745] drop_monitor: Initializing network drop monitor service [ 3.300880] Initializing XFRM netlink socket [ 3.302824] NET: Registered protocol family 10 [ 3.305817] Segment Routing with IPv6 [ 3.306862] NET: Registered protocol family 17 [ 3.308736] mpls_gso: MPLS GSO support [ 3.314915] RAS: Correctable Errors collector initialized. [ 3.316731] AVX version of gcm_enc/dec engaged. [ 3.318156] AES CTR mode by8 optimization enabled [ 3.395297] sched_clock: Marking stable (3395242985, 0)->(4334665231, -939422246) [ 3.398634] registered taskstats version 1 [ 3.400420] Loading compiled-in X.509 certificates [ 3.402500] zswap: loaded using pool lzo/zbud [ 3.430119] Key type big_key registered [ 3.443864] Key type encrypted registered [ 3.445366] ima: No TPM chip found, activating TPM-bypass! [ 3.447292] ima: Allocated hash algorithm: sha1 [ 3.449044] ima: No architecture policies found [ 3.450878] evm: Initialising EVM extended attributes: [ 3.452946] evm: security.selinux [ 3.454130] evm: security.ima [ 3.455314] evm: security.capability [ 3.456574] evm: HMAC attrs: 0x1 [ 3.458970] rtc_cmos 00:05: setting system clock to 2026-08-22 07:16:03 UTC (1787382963) [ 3.465521] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 3.468603] debug: unmapping init [mem 0xffffffffb7582000-0xffffffffb7858fff] [ 3.477177] Write protecting the kernel read-only data: 28672k [ 3.480732] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.483533] debug: unmapping init [mem 0xffffffffb6514000-0xffffffffb65fffff] [ 3.527399] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.536373] systemd[1]: Detected virtualization kvm. [ 3.538517] systemd[1]: Detected architecture x86-64. [ 3.540230] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.568225] systemd[1]: No hostname configured. [ 3.570302] systemd[1]: Set hostname to . [ 3.572549] random: systemd: uninitialized urandom read (16 bytes read) [ 3.575162] systemd[1]: Initializing machine ID from random generator. [ 3.622825] random: ln: uninitialized urandom read (6 bytes read) [ 3.718126] random: systemd: uninitialized urandom read (16 bytes read) [ 3.721101] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.728291] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.732797] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.379242] device-mapper: uevent: version 1.0.3 [ 4.382069] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.217821] random: fast init done [ 5.243986] virtio_net virtio0 ens2: renamed from eth0 [ 5.310336] scsi host0: ata_piix [ 5.340351] scsi host1: ata_piix [ 5.342138] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.344738] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.690530] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.969335] random: crng init done [ 9.970291] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.373435] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.515644] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.768022] SELinux: Disabled at runtime. [ 11.831888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.839314] systemd[1]: Detected virtualization kvm. [ 11.840505] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.348368] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.352572] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.357315] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.361080] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.365191] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.374364] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.388775] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ 12.485802] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.797638] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.126179] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.156996] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.290359] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.298119] EDAC sbridge: Ver: 1.1.2 [ 14.826711] Key type dns_resolver registered [ 15.135600] NFS: Registering the id_resolver key type [ 15.137186] Key type id_resolver registered [ 15.138985] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 40.805983] libcfs: loading out-of-tree module taints kernel. [ 40.898837] Key type ._llcrypt registered [ 40.901218] Key type .llcrypt registered [ 41.016759] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 59.011738] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 60.635664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.653499] alg: No test for adler32 (adler32-zlib) [ 62.173198] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 63.257990] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 65.144152] Key type lgssc registered [ 66.730153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.805043] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.609981] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 131.509301] hrtimer: interrupt took 3291351 ns [ 135.577254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.622973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.957350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 136.999324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.133670] Lustre: lustre-MDT0000: new disk, initializing [ 137.205213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.223817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.095720] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.908188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.023620] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.046280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.049682] Lustre: Skipped 1 previous similar message [ 154.111504] Lustre: lustre-MDT0001: new disk, initializing [ 154.179753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.219258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.233773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.138085] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.280716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.613903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.876902] Lustre: lustre-OST0000: new disk, initializing [ 174.890844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.902550] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.009324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.739382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.747130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.884810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.963541] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.916592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.052595] Lustre: lustre-OST0001: new disk, initializing [ 195.055552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.060174] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.111686] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.794413] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.799468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 202.815699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.875096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.642374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.115597] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 228.425322] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 234.504301] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 239.584742] Lustre: DEBUG MARKER: Client: 2.17.54.114 [ 241.909271] Lustre: DEBUG MARKER: MDS: 2.17.54.114 [ 245.065235] Lustre: DEBUG MARKER: OSS: 2.17.54.114 [ 246.897483] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:20:06 EDT 2026 [ 264.166527] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 265.911075] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.641244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.646049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.660959] Lustre: Skipped 1 previous similar message [ 284.685509] Lustre: Skipped 3 previous similar messages [ 291.916475] Lustre: server umount lustre-MDT0000 complete [ 294.881235] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.919624] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 295.592742] LustreError: 10341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383255 with bad export cookie 2504269889888622067 [ 295.593226] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.606664] LustreError: 10341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 300.000735] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.002044] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.017713] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 300.051797] Lustre: Skipped 2 previous similar messages [ 300.058496] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.065573] Lustre: Skipped 1 previous similar message [ 301.026017] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.040624] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 301.925368] Lustre: server umount lustre-MDT0001 complete [ 306.098508] Lustre: server umount lustre-OST0000 complete [ 309.935678] Lustre: server umount lustre-OST0001 complete [ 316.343691] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 326.795653] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 364.138204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.366518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.398490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.479382] Lustre: lustre-MDT0000: new disk, initializing [ 364.564596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.577599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.889943] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 379.679060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.857819] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.931273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.935915] Lustre: Skipped 1 previous similar message [ 380.052440] Lustre: lustre-MDT0001: new disk, initializing [ 380.225793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.275543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.304923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.956513] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 395.233622] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 401.398360] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 408.149276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.429334] Lustre: lustre-OST0000: new disk, initializing [ 408.436399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.444255] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 408.556732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.722466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.738411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.877489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.614947] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 425.033926] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 429.403961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.562464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 433.813938] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.143185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.805627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.807518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.836033] Lustre: Skipped 1 previous similar message [ 440.840317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.843301] Lustre: Skipped 2 previous similar messages [ 442.163913] Lustre: server umount lustre-OST0000 complete [ 445.931141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.940967] Lustre: Skipped 1 previous similar message [ 445.947535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.041640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.050384] Lustre: Skipped 2 previous similar messages [ 452.119724] Lustre: server umount lustre-MDT0000 complete [ 456.166620] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383416 with bad export cookie 2504269889888623600 [ 456.189107] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 456.210868] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.724190] Lustre: server umount lustre-MDT0001 complete [ 465.539868] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:23:44 (1787383424) [ 487.835108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.393813] LustreError: 19923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.464645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.134923] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 493.543437] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.937402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.847422] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 518.163953] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.809783] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 526.175091] Lustre: Failing over lustre-MDT0000 [ 526.307333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.313400] LustreError: 19920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.319972] Lustre: Skipped 2 previous similar messages [ 526.332921] LustreError: 19920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 528.398562] Lustre: server umount lustre-MDT0000 complete [ 543.720621] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.746070] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 544.740227] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383488/real 1787383488] req@ffff9c19840b0000 x1874206940886400/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787383504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 544.755422] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.292335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.017405] LustreError: 3651:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c1ab4a0d180 x1874206940889088/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 554.326638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.335336] Lustre: Skipped 1 previous similar message [ 558.427579] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 559.600445] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 559.614099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 559.654026] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 568.317851] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 572.831897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.836770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.848820] Lustre: Skipped 1 previous similar message [ 587.232138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 587.424808] Lustre: server umount lustre-MDT0000 complete [ 589.284737] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.292120] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 590.975600] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383551 with bad export cookie 2504269889888625014 [ 590.978883] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 590.988784] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 591.282974] Lustre: server umount lustre-MDT0001 complete [ 599.929713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.514160] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 614.319978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.910890] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 626.380884] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.794505] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 633.948608] Lustre: Failing over lustre-MDT0000 [ 633.968386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.976174] Lustre: Skipped 1 previous similar message [ 633.982628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.988512] Lustre: Skipped 8 previous similar messages [ 634.229677] Lustre: server umount lustre-MDT0000 complete [ 653.139152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.274690] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.548120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.159640] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 658.919506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.927820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.927834] Lustre: Skipped 1 previous similar message [ 658.947701] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 667.525875] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 672.070638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.083422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 672.094434] Lustre: Skipped 1 previous similar message [ 678.415772] Lustre: server umount lustre-MDT0000 complete [ 682.466175] LustreError: 23992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.500237] LustreError: 23992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 683.339169] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383643 with bad export cookie 2504269889888625952 [ 683.350314] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.351765] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 684.238767] Lustre: server umount lustre-MDT0001 complete [ 696.796304] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:27:36 (1787383656) [ 714.925920] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 726.171613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 770.695400] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 781.191995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.429257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 781.450102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 781.514261] Lustre: lustre-MDT0000: new disk, initializing [ 781.567536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.572416] Lustre: Skipped 3 previous similar messages [ 781.585228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 785.867604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 797.477328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.619767] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 797.919557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 802.760245] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 813.208555] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.079373] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 828.881657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.191918] Lustre: 31608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 831.244107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 831.311956] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 831.424579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 831.452420] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 836.316781] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 846.132564] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 851.785902] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 872.418481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.425770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.435714] LustreError: Skipped 2 previous similar messages [ 872.461913] Lustre: Skipped 1 previous similar message [ 872.465221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.488044] Lustre: Skipped 3 previous similar messages [ 877.647706] Lustre: server umount lustre-OST0000 complete [ 887.778760] LustreError: 29651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.800178] LustreError: 29651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 887.899837] Lustre: server umount lustre-MDT0000 complete [ 891.692622] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383851 with bad export cookie 2504269889888626358 [ 891.704274] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.262843] Lustre: server umount lustre-MDT0001 complete [ 903.925647] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 906.744239] Key type lgssc unregistered [ 907.071413] LNet: 33585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.080350] LNetError: 33585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.097167] LNet: Removed LNI 192.168.202.110@tcp [ 908.112287] Key type .llcrypt unregistered [ 908.115951] Key type ._llcrypt unregistered [ 932.061750] Key type ._llcrypt registered [ 932.063543] Key type .llcrypt registered [ 932.127311] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 933.349308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.392538] alg: No test for adler32 (adler32-zlib) [ 934.467975] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 934.702774] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 936.384681] Key type lgssc registered [ 937.591446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.334600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.374356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.234217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.673129] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 957.421336] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.436901] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 962.246759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.529646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 962.903892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.213909] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 978.635849] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 985.644835] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 994.493869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.240706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.427727] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1007.550629] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1013.224795] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1022.970688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1031.684772] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:33:11 (1787383991) [ 1033.680727] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1035.700261] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1042.988095] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:33:22 (1787384002) [ 1052.360823] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:33:32 (1787384012) [ 1053.282933] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1063.294610] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1063.299873] Lustre: Skipped 2 previous similar messages [ 1065.641332] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:33:45 (1787384025) [ 1067.762584] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1067.768936] Lustre: Skipped 2 previous similar messages [ 1076.256543] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1076.262231] Lustre: Skipped 4 previous similar messages [ 1092.044859] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:34:11 (1787384051) [ 1095.168574] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1098.508598] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1108.013849] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:34:27 (1787384067) [ 1110.255707] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1110.266308] Lustre: Skipped 6 previous similar messages [ 1124.118727] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:34:43 (1787384083) [ 1145.833129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.846908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.847722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.868666] Lustre: Skipped 1 previous similar message [ 1150.948463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.954135] Lustre: Skipped 1 previous similar message [ 1155.043963] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1155.324569] Lustre: server umount lustre-OST0000 complete [ 1161.185925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.187425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.203775] Lustre: Skipped 2 previous similar messages [ 1161.212732] Lustre: Skipped 1 previous similar message [ 1165.026848] Lustre: server umount lustre-MDT0000 complete [ 1166.307130] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.338619] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1169.079680] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384129 with bad export cookie 3250284088036853086 [ 1169.087989] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.093517] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.459789] Lustre: server umount lustre-MDT0001 complete [ 1181.654772] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1184.432650] Key type lgssc unregistered [ 1184.687719] LNet: 42848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.695420] LNetError: 42848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.712844] LNet: Removed LNI 192.168.202.110@tcp [ 1185.461763] Key type .llcrypt unregistered [ 1185.462989] Key type ._llcrypt unregistered [ 1203.994756] Key type ._llcrypt registered [ 1203.997597] Key type .llcrypt registered [ 1204.097484] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 1221.192343] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1222.645799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.799934] alg: No test for adler32 (adler32-zlib) [ 1223.776824] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1223.962746] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1225.624540] Key type lgssc registered [ 1226.798545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.186792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.232618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.610241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1267.671323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1267.801543] Lustre: lustre-MDT0000: new disk, initializing [ 1267.943141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.997845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1273.642185] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1283.960711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.071556] Lustre: 45997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1284.119518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1284.123213] Lustre: Skipped 1 previous similar message [ 1284.180582] Lustre: lustre-MDT0001: new disk, initializing [ 1284.277445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1284.335882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1284.340904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1289.026329] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1299.330023] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.328918] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1314.696985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.945964] Lustre: lustre-OST0000: new disk, initializing [ 1314.949134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1314.953875] Lustre: 47972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1315.035449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.134823] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1324.128128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1324.144257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1324.237631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1332.716135] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1338.524566] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1338.826758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1344.424150] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1344.633844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1350.121332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1350.122251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.128779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.143850] LustreError: Skipped 1 previous similar message [ 1350.167534] Lustre: Skipped 1 previous similar message [ 1352.971757] Lustre: server umount lustre-OST0000 complete [ 1360.354519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.375510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.386803] Lustre: Skipped 2 previous similar messages [ 1365.940404] Lustre: server umount lustre-MDT0000 complete [ 1370.316663] LustreError: 45991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384330 with bad export cookie 7157116787685122026 [ 1370.324461] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1370.327459] LustreError: 45991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1370.990822] Lustre: server umount lustre-MDT0001 complete [ 1382.200112] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:39:01 (1787384341) [ 1389.704223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.083226] LustreError: 50432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.114220] LustreError: 50432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1390.173933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.860297] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1395.172950] LustreError: 50433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.300473] LustreError: 50432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.739824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.127372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1408.567782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1415.908733] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1421.977958] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1429.469229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.972769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.466654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1444.012422] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1449.065765] LustreError: 52961:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1450.360046] Lustre: Modifying parameter general.timeout=1 in log params [ 1451.561557] Lustre: Modifying parameter general.timeout=2 in log params [ 1453.821176] Lustre: Modifying parameter general.timeout=4 in log params [ 1453.825955] Lustre: Skipped 1 previous similar message [ 1458.507961] Lustre: Modifying parameter general.timeout=8 in log params [ 1458.512712] Lustre: Skipped 3 previous similar messages [ 1467.076220] Lustre: Modifying parameter general.timeout=16 in log params [ 1467.082651] Lustre: Skipped 7 previous similar messages [ 1484.044798] Lustre: Modifying parameter general.timeout=33 in log params [ 1484.074434] Lustre: Skipped 16 previous similar messages [ 1516.050469] Lustre: Modifying parameter general.timeout=65 in log params [ 1516.060662] Lustre: Skipped 31 previous similar messages [ 1581.012297] Lustre: Modifying parameter general.timeout=127 in log params [ 1581.021661] Lustre: Skipped 61 previous similar messages [ 1595.446645] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:42:35 (1787384555) [ 1599.119871] Lustre: 59911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1600.800924] Lustre: 60009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1604.721569] Lustre: 60223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1604.733760] Lustre: 60223:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1638.621969] Lustre: 61790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1638.632752] Lustre: 61790:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1638.664540] LustreError: 61790:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1643.050669] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:43:22 (1787384602) [ 1662.435450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.441385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.470620] Lustre: Skipped 2 previous similar messages [ 1667.535295] Lustre: server umount lustre-MDT0000 complete [ 1667.558441] LustreError: 51160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.592994] LustreError: 51160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1672.453271] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384632 with bad export cookie 7157116787685122908 [ 1672.462429] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.466734] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1672.677033] LustreError: 50429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.681659] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.703407] Lustre: Skipped 1 previous similar message [ 1672.787299] Lustre: server umount lustre-MDT0001 complete [ 1679.768807] Lustre: server umount lustre-OST0000 complete [ 1709.497761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.888705] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1709.916698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1709.934032] Lustre: Skipped 25 previous similar messages [ 1710.217223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.923949] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1724.128356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.201630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1724.224636] Lustre: 64323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1729.175443] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1737.950110] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.310259] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1753.434200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.753774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1754.322572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.327199] Lustre: Skipped 1 previous similar message [ 1761.588150] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1770.895751] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1773.524165] Lustre: 66891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1785.318700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.336128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.348702] Lustre: Skipped 3 previous similar messages [ 1790.450650] Lustre: server umount lustre-OST0000 complete [ 1795.554288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.556097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.585284] Lustre: Skipped 2 previous similar messages [ 1795.607325] Lustre: Skipped 1 previous similar message [ 1800.873402] Lustre: server umount lustre-MDT0000 complete [ 1805.109207] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384765 with bad export cookie 7157116787685126086 [ 1805.125519] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.440799] Lustre: server umount lustre-MDT0001 complete [ 1819.352916] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1822.467576] Key type lgssc unregistered [ 1822.841739] LNet: 68103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.850160] LNetError: 68103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.872273] LNet: Removed LNI 192.168.202.110@tcp [ 1823.992187] Key type .llcrypt unregistered [ 1823.994945] Key type ._llcrypt unregistered [ 1854.494816] Key type ._llcrypt registered [ 1854.499234] Key type .llcrypt registered [ 1854.621580] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1856.284028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.314977] alg: No test for adler32 (adler32-zlib) [ 1857.418288] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1857.642456] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1859.320243] Key type lgssc registered [ 1860.774482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1868.346170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1868.360742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.886341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.444729] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1879.011110] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.029220] LustreError: 69519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1884.131058] LustreError: 69518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1884.734896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.253560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1890.684425] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1902.861588] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.800633] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.359729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.877850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.054471] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1939.587515] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1948.358976] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1959.907986] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.911179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.933580] Lustre: Skipped 1 previous similar message [ 1964.335316] Lustre: server umount lustre-MDT0000 complete [ 1965.026864] LustreError: 69518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.436635] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384928 with bad export cookie 7728452241980400862 [ 1968.442811] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1968.444120] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.735841] Lustre: server umount lustre-MDT0001 complete [ 1984.129504] Lustre: server umount lustre-OST0000 complete [ 2016.309394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.773586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2016.828718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2017.200111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.352733] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2032.236740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.337201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2032.424658] Lustre: 74393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2032.833383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.854467] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2045.949883] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2052.938254] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2059.262645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.455362] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2059.772234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.053568] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2073.882789] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.770028] Lustre: 76969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2077.837288] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2083.812036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2083.825192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.841929] Lustre: Skipped 1 previous similar message [ 2083.858289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.860913] Lustre: Skipped 2 previous similar messages [ 2088.846763] Lustre: server umount lustre-OST0000 complete [ 2094.050716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.065539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.089062] Lustre: Skipped 1 previous similar message [ 2094.098811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.106422] Lustre: Skipped 2 previous similar messages [ 2099.683707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.686413] Lustre: Skipped 1 previous similar message [ 2104.803448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.815200] Lustre: Skipped 1 previous similar message [ 2106.337750] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2106.574511] Lustre: server umount lustre-MDT0000 complete [ 2109.927735] LustreError: 74401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.956764] LustreError: 74401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2110.491637] LustreError: 74387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385070 with bad export cookie 7728452241980401702 [ 2110.500058] LustreError: 74387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2110.500713] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.257045] Lustre: server umount lustre-MDT0001 complete [ 2123.540824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2126.280909] Key type lgssc unregistered [ 2126.684425] LNet: 78187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.698794] LNetError: 78187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.721533] LNet: Removed LNI 192.168.202.110@tcp [ 2127.809337] Key type .llcrypt unregistered [ 2127.813178] Key type ._llcrypt unregistered [ 2153.388254] Key type ._llcrypt registered [ 2153.389430] Key type .llcrypt registered [ 2153.517165] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2154.574245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.681683] alg: No test for adler32 (adler32-zlib) [ 2155.973767] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2156.267042] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2157.920299] Key type lgssc registered [ 2159.445782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.650064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.691989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.295695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.157478] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2180.578422] LustreError: 79602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.599595] LustreError: 79602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2182.591835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.102462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.693380] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2195.895772] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2202.218759] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2210.869953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2211.647262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.979910] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2228.178990] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2234.983630] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2236.097530] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2237.254468] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2239.698515] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2239.705726] Lustre: Skipped 1 previous similar message [ 2244.549692] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2244.553553] Lustre: Skipped 4 previous similar messages [ 2253.556532] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2253.559572] Lustre: Skipped 8 previous similar messages [ 2270.002475] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2270.011699] Lustre: Skipped 16 previous similar messages [ 2302.154694] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2302.166431] Lustre: Skipped 29 previous similar messages [ 2349.833974] Lustre: 87711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2352.863742] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:55:12 (1787385312) [ 2354.808359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2356.771799] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:55:16 (1787385316) [ 2364.385656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2364.394075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.405306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.409188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2365.422396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.431437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.459408] Lustre: Skipped 1 previous similar message [ 2368.253413] Lustre: server umount lustre-OST0000 complete [ 2375.651472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.683734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.710446] Lustre: Skipped 1 previous similar message [ 2379.168661] Lustre: server umount lustre-MDT0000 complete [ 2380.778046] LustreError: 79603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.808351] LustreError: 79603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2382.716763] LustreError: 79583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385342 with bad export cookie 11401945427807947443 [ 2382.720636] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.727553] LustreError: 79583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2383.167837] Lustre: server umount lustre-MDT0001 complete [ 2393.175924] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2395.719656] Key type lgssc unregistered [ 2395.977782] LNet: 88959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.985822] LNetError: 88959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.035236] LNet: Removed LNI 192.168.202.110@tcp [ 2398.036665] Key type .llcrypt unregistered [ 2398.038768] Key type ._llcrypt unregistered [ 2402.632196] Key type ._llcrypt registered [ 2402.634305] Key type .llcrypt registered [ 2402.743700] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2406.979893] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules [ 2408.099028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.127053] alg: No test for adler32 (adler32-zlib) [ 2409.192269] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2409.443912] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2411.096212] Key type lgssc registered [ 2412.191727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.284360] LustreError: 90163:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2430.954250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2435.510195] LustreError: 90163:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2440.709040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.743590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.403722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.643257] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2452.964729] LustreError: 91187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.997081] LustreError: 91187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2456.883251] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:56:56 (1787385416) [ 2458.085764] LustreError: 91188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.781711] LustreError: 91219:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2461.788638] LustreError: 91219:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2461.794249] LustreError: 91219:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2462.506042] Lustre: server umount lustre-MDT0000 complete [ 2476.243206] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2478.712971] Key type lgssc unregistered [ 2478.946535] LNet: 92714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.953938] LNetError: 92714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.973874] LNet: Removed LNI 192.168.202.110@tcp [ 2479.720274] Key type .llcrypt unregistered [ 2479.724231] Key type ._llcrypt unregistered [ 2498.151673] Key type ._llcrypt registered [ 2498.153603] Key type .llcrypt registered [ 2498.241354] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2499.325409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.461283] alg: No test for adler32 (adler32-zlib) [ 2500.553853] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2500.775473] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2502.464200] Key type lgssc registered [ 2503.601793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.962897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.982696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.500470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.404174] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2523.633462] LustreError: 93933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.657752] LustreError: 93933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2528.737852] LustreError: 93934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.649353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.596454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.779814] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2543.487635] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2549.293436] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2555.447432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.930352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.174774] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2568.961071] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2626.563980] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:59:46 (1787385586) [ 2628.125154] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2629.991287] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:59:49 (1787385589) [ 2636.792748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.802748] Lustre: Skipped 1 previous similar message [ 2636.804803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.811778] Lustre: Skipped 2 previous similar messages [ 2640.571825] Lustre: server umount lustre-MDT0000 complete [ 2641.891197] LustreError: 96239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.914116] LustreError: 96239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2644.352907] LustreError: 93914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385604 with bad export cookie 17876891986141309485 [ 2644.357830] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2644.367168] LustreError: 93914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2644.812348] Lustre: server umount lustre-MDT0001 complete [ 2659.488968] Lustre: server umount lustre-OST0000 complete [ 2672.541979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.969349] LustreError: 98232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.169512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.842357] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2678.243554] LustreError: 98233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.255478] LustreError: 98233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2686.766109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.637256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.814695] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2699.999966] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2706.237907] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2719.053881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2719.279985] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2719.286355] LustreError: 98222:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2719.307956] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2719.314372] LustreError: 100328:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2719.319892] LustreError: 100328:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2719.326570] LustreError: 100328:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2719.410431] Lustre: server umount lustre-OST0000 complete [ 2719.412866] LustreError: 100328:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2729.684633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.992969] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2729.998147] LustreError: 98219:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2730.022897] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2730.031216] LustreError: 100795:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2730.037328] LustreError: 100795:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2730.042336] LustreError: 100795:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2730.158150] Lustre: server umount lustre-OST0000 complete [ 2730.164126] LustreError: 100795:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2740.530632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.700784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2740.719214] Lustre: Found index 0 for lustre-OST0000, updating log [ 2740.727590] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2740.811254] Lustre: lustre-OST0000: new disk, initializing [ 2740.816189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2740.825501] Lustre: 101330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2741.121465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.507266] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2753.216992] Lustre: Failing over lustre-OST0000 [ 2753.389162] Lustre: server umount lustre-OST0000 complete [ 2758.627936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.630895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.647819] Lustre: Skipped 1 previous similar message [ 2759.719991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.734606] Lustre: Skipped 2 previous similar messages [ 2764.771211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.778554] Lustre: Skipped 1 previous similar message [ 2768.979065] Lustre: server umount lustre-MDT0000 complete [ 2769.897159] LustreError: 98229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.945949] LustreError: 98229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2773.536641] LustreError: 98973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385733 with bad export cookie 17876891986141312229 [ 2773.538189] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.548027] LustreError: 98973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2773.905220] Lustre: server umount lustre-MDT0001 complete [ 2786.781458] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:02:26 (1787385746) [ 2814.002507] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2826.265654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.009851] LustreError: 103700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.213880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.322524] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2843.074182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.195985] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2851.125457] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2858.735894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2860.233897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.255492] Lustre: Skipped 1 previous similar message [ 2860.276525] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2868.471969] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2868.508657] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2868.571911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2869.479073] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2872.626419] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2872.659622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2872.689762] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2872.690515] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2872.723408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2881.252612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.417408] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2881.563176] Lustre: lustre-OST0001: new disk, initializing [ 2881.569537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2881.587394] Lustre: 106017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.772578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2887.785089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2887.959779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2889.384459] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2899.054443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2910.309257] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2915.809245] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2915.822857] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.841459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2918.889823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.915289] Lustre: Skipped 1 previous similar message [ 2921.699216] Lustre: server umount lustre-MDT0001 complete [ 2923.678766] LustreError: 103729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.703826] LustreError: 103729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2931.704085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.828460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2931.832141] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2931.881031] Lustre: 103689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2932.932350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.944831] Lustre: Skipped 1 previous similar message [ 2933.488000] LustreError: 107559:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2933.498974] LustreError: 107559:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2933.512985] LustreError: 103692:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2933.648630] LustreError: 107569:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.110@tcp: cfg command failed: rc = -17 [ 2933.658751] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2933.658751] [ 2938.157325] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2938.363824] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2949.720816] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:05:09 (1787385909) [ 2970.422818] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2991.434603] Lustre: 109637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3020.662680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3026.275379] Lustre: 110773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3125.787121] Lustre: server umount lustre-MDT0000 complete [ 3126.756126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3126.763234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.785558] LustreError: 103700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.811218] LustreError: 103700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3129.878844] LustreError: 104443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386089 with bad export cookie 17876891986141313230 [ 3129.879678] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.891662] LustreError: 104443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3130.305639] Lustre: server umount lustre-MDT0001 complete [ 3145.009872] Lustre: server umount lustre-OST0000 complete [ 3148.240421] Lustre: 93244:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386092/real 1787386092] req@ffff9c1abaf9c380 x1874209496488064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787386108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3148.311235] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.341753] Lustre: Skipped 3 previous similar messages [ 3149.849408] Lustre: server umount lustre-OST0001 complete [ 3156.012553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3167.455558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3177.090859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3189.425240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3215.447586] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3231.852279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.875766] Lustre: lustre-MDT0000: reset Object Index mappings [ 3232.900352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.159963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3249.654391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.677314] Lustre: lustre-MDT0001: reset Object Index mappings [ 3255.322373] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3259.110812] Lustre: 115873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3267.151409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.273169] LustreError: 116227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.296539] LustreError: 116227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3275.314412] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3275.765507] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3277.307513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3285.962635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3292.156641] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3292.162626] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3295.183071] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3304.757753] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3309.126791] Lustre: 117751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3375.754405] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:12:14 (1787386334) [ 3403.752259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3403.761601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.779607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.784316] Lustre: Skipped 4 previous similar messages [ 3407.789667] Lustre: server umount lustre-MDT0000 complete [ 3412.006406] LustreError: 114706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386372 with bad export cookie 17876891986141360319 [ 3412.025041] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3418.657535] Lustre: server umount lustre-MDT0001 complete [ 3429.716757] Lustre: server umount lustre-OST0000 complete [ 3448.288177] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3448.637735] Lustre: server umount lustre-OST0001 complete [ 3457.058988] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3469.476457] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3509.190424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.479921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3509.547235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3509.611149] Lustre: lustre-MDT0000: new disk, initializing [ 3509.707788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.711950] Lustre: Skipped 3 previous similar messages [ 3509.735613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3514.888568] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3522.027576] Lustre: server umount lustre-MDT0000 complete [ 3533.490617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.831574] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3533.848635] Lustre: 122023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3533.859952] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3539.052300] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3544.774526] Lustre: server umount lustre-MDT0000 complete [ 3554.537727] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:15:14 (1787386514) [ 3556.119655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3557.949191] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:15:17 (1787386517) [ 3576.378589] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3587.068243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.587287] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3603.684320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.985489] Lustre: 124075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.097828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3604.100918] Lustre: Skipped 1 previous similar message [ 3604.223338] Lustre: lustre-MDT0001: new disk, initializing [ 3604.552052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3604.589351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3610.062740] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3615.164042] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3621.999805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.323981] Lustre: lustre-OST0000: new disk, initializing [ 3622.328304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.334082] Lustre: 125651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.814482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3623.833670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3623.951910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3628.572782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3639.332193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3639.466422] Lustre: lustre-OST0001: new disk, initializing [ 3639.469858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3639.475920] Lustre: 126523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3640.912887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3640.925871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3641.025526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3645.973242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3655.912567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3659.766945] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4061.588663] Lustre: 132947:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4061.601727] Lustre: 132947:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4061.611724] Lustre: 132947:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4061.625176] Lustre: 132947:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4061.633874] Lustre: 132947:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4061.648322] Lustre: 132947:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4364.731928] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:28:43 (1787387323) [ 4381.667211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.678219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.720182] Lustre: Skipped 8 previous similar messages [ 4381.721339] Lustre: Skipped 6 previous similar messages [ 4386.787577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.799841] Lustre: Skipped 4 previous similar messages [ 4388.124840] Lustre: server umount lustre-MDT0000 complete [ 4391.913660] LustreError: 125645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.936774] LustreError: 125645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4392.196242] LustreError: 132960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387352 with bad export cookie 17876891986141431453 [ 4392.205710] LustreError: 132960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4392.214637] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4397.037923] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.060259] Lustre: Skipped 2 previous similar messages [ 4397.064149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4397.071021] Lustre: Skipped 1 previous similar message [ 4398.881564] Lustre: server umount lustre-MDT0001 complete [ 4408.380549] Lustre: server umount lustre-OST0000 complete [ 4418.903237] Lustre: server umount lustre-OST0001 complete [ 4427.860861] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4440.220268] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4478.840034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.147562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4479.182928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4479.291813] Lustre: lustre-MDT0000: new disk, initializing [ 4479.446171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.449402] Lustre: Skipped 5 previous similar messages [ 4479.463623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.166147] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4495.631947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.838368] Lustre: 136006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4495.982627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4495.988692] Lustre: Skipped 1 previous similar message [ 4496.147562] Lustre: lustre-MDT0001: new disk, initializing [ 4496.355103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4496.376617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4501.688466] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4514.257346] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.581867] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4529.294936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4529.799642] Lustre: lustre-OST0000: new disk, initializing [ 4529.804361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4529.813722] Lustre: 137975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4530.936628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4530.962814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4531.119720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.117729] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4550.395276] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4563.570923] Lustre: lustre-MDD0000: changelog on [ 4567.578902] Lustre: lustre-MDD0001: changelog on [ 6704.809815] Lustre: lustre-MDD0001: changelog off [ 6707.430570] Lustre: 136013:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6708.493549] Lustre: lustre-MDD0000: changelog off [ 6717.411084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.412018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.421732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6717.427977] LustreError: Skipped 1 previous similar message [ 6717.439732] Lustre: Skipped 1 previous similar message [ 6722.548784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.563957] Lustre: Skipped 1 previous similar message [ 6723.708522] Lustre: server umount lustre-OST0000 complete [ 6732.773878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.789139] Lustre: Skipped 1 previous similar message [ 6732.796607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.815850] Lustre: Skipped 1 previous similar message [ 6735.612327] Lustre: server umount lustre-MDT0000 complete [ 6737.889164] LustreError: 136016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.923788] LustreError: 136016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6741.096502] LustreError: 135996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389701 with bad export cookie 17876891986142253344 [ 6741.108685] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6741.118488] LustreError: 135996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6741.899583] Lustre: server umount lustre-MDT0001 complete [ 6757.277944] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 6761.168041] Key type lgssc unregistered [ 6761.557264] LNet: 147159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6761.571075] LNetError: 147159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6761.606093] LNet: Removed LNI 192.168.202.110@tcp [ 6762.458929] Key type .llcrypt unregistered [ 6762.460841] Key type ._llcrypt unregistered [ 6791.381090] Key type ._llcrypt registered [ 6791.383624] Key type .llcrypt registered [ 6791.606668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 6810.953686] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 6812.727403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6812.821906] alg: No test for adler32 (adler32-zlib) [ 6814.075945] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 6814.461184] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 6816.232199] Key type lgssc registered [ 6817.757757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6868.356798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6868.369534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.874777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6869.924539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6870.033499] Lustre: lustre-MDT0000: new disk, initializing [ 6870.202790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.236296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6876.233558] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6890.103444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.338197] Lustre: 150364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6890.415205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6890.418796] Lustre: Skipped 1 previous similar message [ 6890.547208] Lustre: lustre-MDT0001: new disk, initializing [ 6890.674898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6890.743935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6890.757531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6897.317162] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6909.733702] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6917.383857] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6926.394303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.965461] Lustre: lustre-OST0000: new disk, initializing [ 6926.974513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6926.985161] Lustre: 152336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6927.085090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.700162] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6936.211516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6936.222578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6936.403764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6947.806812] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6956.157520] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module obdecho/obdecho [ 6958.543964] LustreError: 153340:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6958.552675] LustreError: 153340:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6961.114810] LustreError: 153437:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6982.626663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.642413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.664120] Lustre: Skipped 2 previous similar messages [ 6987.108836] Lustre: server umount lustre-MDT0000 complete [ 6991.895518] LustreError: 151161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389951 with bad export cookie 949743452600579931 [ 6991.899044] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.910314] LustreError: 151161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6992.216969] Lustre: server umount lustre-MDT0001 complete [ 7008.309388] Lustre: server umount lustre-OST0000 complete [ 7038.759760] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 7057.467648] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7106.687696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.947678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7107.017616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7107.086233] Lustre: lustre-MDT0000: new disk, initializing [ 7107.177917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.197109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7112.561780] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7127.545838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.746486] Lustre: 157181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7127.835174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7127.844303] Lustre: Skipped 1 previous similar message [ 7127.934158] Lustre: lustre-MDT0001: new disk, initializing [ 7128.073664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.124517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7128.134713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7135.170868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7147.066513] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7155.195792] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7164.916317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.438848] Lustre: lustre-OST0000: new disk, initializing [ 7165.453441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7165.466234] Lustre: 159150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7165.662924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.269847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7174.273366] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7174.282217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7174.437173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7184.774130] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7190.491210] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7190.789773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7195.995788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7196.247187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7200.227130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7200.229495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.244755] LustreError: Skipped 1 previous similar message [ 7200.253886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7200.256685] Lustre: Skipped 1 previous similar message [ 7200.266082] Lustre: Skipped 2 previous similar messages [ 7204.635707] Lustre: server umount lustre-OST0000 complete [ 7209.954563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7209.961088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.976882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.979564] Lustre: Skipped 1 previous similar message [ 7215.596556] LustreError: 157192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.611585] LustreError: 157192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7215.774149] Lustre: server umount lustre-MDT0000 complete [ 7220.145981] LustreError: 157172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390180 with bad export cookie 949743452600580890 [ 7220.157433] LustreError: 157172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7220.172595] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7220.545908] Lustre: server umount lustre-MDT0001 complete [ 7224.331699] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:16:24 (1787390184) [ 7226.415524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7228.610852] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:16:28 (1787390188) [ 7237.420660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.841644] LustreError: 161563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.946230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.234922] LustreError: 161564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.611683] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7247.331031] LustreError: 161563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.451699] LustreError: 161564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.238241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.009371] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7270.892654] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7277.685594] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.522289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.940346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.947645] Lustre: Skipped 1 previous similar message [ 7293.828832] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7303.894777] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7317.992199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.007171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7318.014123] Lustre: Skipped 3 previous similar messages [ 7318.016054] Lustre: Skipped 3 previous similar messages [ 7322.560043] Lustre: server umount lustre-MDT0000 complete [ 7323.109893] LustreError: 161563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.144237] LustreError: 161563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7326.981025] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390287 with bad export cookie 949743452600581772 [ 7326.988707] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7326.996454] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7328.228102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.250726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7328.256164] Lustre: Skipped 2 previous similar messages [ 7333.716606] Lustre: server umount lustre-MDT0001 complete [ 7341.107076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7356.598895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.833445] LustreError: 165429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.854153] LustreError: 165429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7356.921642] LustreError: 165424:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7356.935151] Lustre: 165424:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7390.881763] LustreError: 165431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.904455] LustreError: 165431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 7418.336596] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7418.347218] Lustre: 165424:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7456.419511] LustreError: 165431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7456.440273] LustreError: 165431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7459.328995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7464.444256] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7464.778188] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7475.421542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.974380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7497.191107] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd2e2aa36002d7e5 [ 7497.218627] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [ 7502.835546] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7503.048073] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7511.697595] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7514.153677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7520.345929] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7523.175822] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7545.206154] Lustre: server umount lustre-MDT0000 complete [ 7548.912543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7548.923416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.945702] Lustre: Skipped 2 previous similar messages [ 7549.995507] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390510 with bad export cookie 949743452600588261 [ 7550.002020] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.017540] LustreError: 163409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7550.707416] Lustre: server umount lustre-MDT0001 complete [ 7566.012494] Lustre: server umount lustre-OST0000 complete [ 7591.072087] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 7603.637795] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7645.432217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.662078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.706613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7645.810112] Lustre: lustre-MDT0000: new disk, initializing [ 7645.921373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.926486] Lustre: Skipped 1 previous similar message [ 7645.969108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7650.751651] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7663.184279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.387709] Lustre: 170533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7663.789818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7669.785604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7680.470647] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7687.873084] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7696.424457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7696.766350] Lustre: lustre-OST0000: new disk, initializing [ 7696.773886] Lustre: Skipped 1 previous similar message [ 7696.780154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7696.782611] Lustre: Skipped 2 previous similar messages [ 7696.789580] Lustre: 172502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7704.175724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7704.186212] Lustre: Skipped 1 previous similar message [ 7704.200422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7704.315261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7704.678513] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7719.173391] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7724.587966] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7724.807626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7730.029468] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7730.267644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7735.280897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.283625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.294978] LustreError: Skipped 1 previous similar message [ 7735.295845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.295849] Lustre: Skipped 2 previous similar messages [ 7735.344095] Lustre: Skipped 1 previous similar message [ 7738.995979] Lustre: server umount lustre-OST0000 complete [ 7745.518406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7750.210614] Lustre: server umount lustre-MDT0000 complete [ 7750.625745] LustreError: 171378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7750.655124] LustreError: 171378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7755.634058] LustreError: 170528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390715 with bad export cookie 949743452600605068 [ 7755.646416] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7755.647504] LustreError: 170528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7756.631278] Lustre: server umount lustre-MDT0001 complete [ 7769.500734] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:25:29 (1787390729) [ 7778.446708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.651841] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7794.103727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.141500] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7810.130420] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7817.219554] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7823.898039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.572373] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7841.066603] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7852.099739] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:26:51 (1787390811) [ 7860.201076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7860.204301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.217566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.217575] Lustre: Skipped 3 previous similar messages [ 7860.218218] LustreError: Skipped 1 previous similar message [ 7860.315017] Lustre: Skipped 3 previous similar messages [ 7862.587936] Lustre: server umount lustre-OST0000 complete [ 7873.393429] Lustre: server umount lustre-MDT0000 complete [ 7878.501697] LustreError: 174898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390838 with bad export cookie 949743452600605950 [ 7878.506333] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7878.514131] LustreError: 174898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7879.128614] Lustre: server umount lustre-MDT0001 complete [ 7892.514721] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7895.665960] Key type lgssc unregistered [ 7896.089670] LNet: 178652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7896.095123] LNetError: 178652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7896.115823] LNet: Removed LNI 192.168.202.110@tcp [ 7897.257640] Key type .llcrypt unregistered [ 7897.260194] Key type ._llcrypt unregistered [ 7919.244970] Key type ._llcrypt registered [ 7919.248277] Key type .llcrypt registered [ 7919.459321] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7921.196161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7921.598704] alg: No test for adler32 (adler32-zlib) [ 7923.021474] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7923.566343] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7925.400225] Key type lgssc registered [ 7926.846571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7936.038201] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7936.066209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.584388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7956.705713] LustreError: 179896:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [ 7961.825340] LustreError: 179896:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7961.847232] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7961.853378] LustreError: MGC192.168.202.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7961.860658] LustreError: 179896:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7961.867311] LustreError: 179896:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7961.876170] LustreError: 179896:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7961.947212] Lustre: server umount lustre-OST0000 complete [ 7961.951341] LustreError: 179896:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7971.653427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.309391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.356278] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7983.587292] LustreError: 180396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.611744] LustreError: 180396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7988.705985] LustreError: 180395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.216479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.615839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7995.638531] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8003.779991] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8011.588541] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8019.859123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.206384] LustreError: 182213:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8020.216931] Lustre: 182213:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8020.399645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8027.174654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8035.277988] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8040.947852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.975408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.986340] Lustre: Skipped 2 previous similar messages [ 8041.000920] Lustre: Skipped 2 previous similar messages [ 8043.791379] LustreError: 182973:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000c5905462) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8044.137304] Lustre: server umount lustre-MDT0000 complete [ 8046.052413] LustreError: 181393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.076851] LustreError: 181393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8048.176408] LustreError: 182240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391008 with bad export cookie 8633398008590721338 [ 8048.182101] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.193986] LustreError: 182240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8048.890370] Lustre: server umount lustre-MDT0001 complete [ 8064.070918] Lustre: server umount lustre-OST0000 complete [ 8074.688682] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8086.287748] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8128.205557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.337601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8128.368470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8128.446774] Lustre: lustre-MDT0000: new disk, initializing [ 8128.589109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.619111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8134.038522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8144.867971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.987332] Lustre: 185737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8145.053396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8145.058163] Lustre: Skipped 1 previous similar message [ 8145.128298] Lustre: lustre-MDT0001: new disk, initializing [ 8145.232931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.284875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8145.308985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8150.351618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8161.508295] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8168.888698] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8176.815972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8177.101959] Lustre: lustre-OST0000: new disk, initializing [ 8177.107645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8177.112797] Lustre: 187707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8177.184493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.378068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8183.386205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8183.534538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8184.359333] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8196.876564] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8203.855788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8204.224804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8209.320738] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8209.508699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8214.504994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8214.509881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8214.522322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8214.530254] Lustre: Skipped 1 previous similar message [ 8219.618188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8219.629523] Lustre: Skipped 2 previous similar messages [ 8224.744415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8224.752901] Lustre: Skipped 1 previous similar message [ 8226.273631] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8226.615421] Lustre: server umount lustre-OST0000 complete [ 8232.417650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8232.431576] LustreError: Skipped 1 previous similar message [ 8232.445852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.475599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.982824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.070047] Lustre: server umount lustre-MDT0000 complete [ 8240.103478] LustreError: 186598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.131110] LustreError: 186598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8241.137238] LustreError: 185730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391201 with bad export cookie 8633398008590722052 [ 8241.141443] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8241.154059] LustreError: 185730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8241.712863] Lustre: server umount lustre-MDT0001 complete [ 8252.993463] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:33:32 (1787391212) [ 8274.616801] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8277.434894] Key type lgssc unregistered [ 8277.728747] LNet: 190658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8277.736411] LNetError: 190658:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8277.760190] LNet: Removed LNI 192.168.202.110@tcp [ 8278.456354] Key type .llcrypt unregistered [ 8278.457872] Key type ._llcrypt unregistered [ 8304.132397] Key type ._llcrypt registered [ 8304.141508] Key type .llcrypt registered [ 8304.333785] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8305.503922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8305.582092] alg: No test for adler32 (adler32-zlib) [ 8306.662969] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8306.947651] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8308.600975] Key type lgssc registered [ 8310.058762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8318.375440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8318.401473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8319.899853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8325.437672] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8329.202076] LustreError: 191876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.229690] LustreError: 191876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8334.320160] LustreError: 191877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.020477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.462637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.955406] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8353.471169] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8360.210362] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8369.581473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8369.731539] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8369.736384] LustreError: 193740:0:(llog.c:1696:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8369.741702] Lustre: 193740:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8369.841482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.510722] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8385.346098] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8388.582325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.592368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.607353] Lustre: Skipped 2 previous similar messages [ 8393.699511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8393.715350] Lustre: Skipped 2 previous similar messages [ 8394.801088] Lustre: server umount lustre-MDT0000 complete [ 8398.843603] LustreError: 191877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.877617] LustreError: 191877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8399.607267] LustreError: 191859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391359 with bad export cookie 16518419890281103806 [ 8399.619878] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8399.625411] LustreError: 191859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8400.199667] Lustre: server umount lustre-MDT0001 complete [ 8415.222452] Lustre: server umount lustre-OST0000 complete [ 8426.971927] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8444.614784] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8487.226361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8487.491854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8487.526749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8487.618333] Lustre: lustre-MDT0000: new disk, initializing [ 8487.728442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.760029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8493.127028] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8505.922148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.043829] Lustre: 197292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8506.107553] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8506.110837] Lustre: Skipped 1 previous similar message [ 8506.113906] LustreError: 198077:0:(llog.c:1696:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8506.120706] LustreError: 198077:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8506.125964] Lustre: 198077:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8506.149445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8506.154699] Lustre: Skipped 1 previous similar message [ 8506.244960] Lustre: lustre-MDT0001: new disk, initializing [ 8506.376229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8506.490542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8506.507861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8511.958644] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8522.658645] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8529.274100] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8537.292156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8537.686812] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8537.692241] Lustre: Skipped 1 previous similar message [ 8537.694426] LustreError: 199238:0:(llog.c:1696:llog_backup()) MGC192.168.202.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8537.704440] LustreError: 199238:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8537.711182] Lustre: 199238:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8537.721233] Lustre: 199238:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8537.749837] Lustre: lustre-OST0000: new disk, initializing [ 8537.756656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8537.771904] Lustre: 199261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8537.899401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.910095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8544.924316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8545.216559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8547.210568] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8559.131171] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8564.175822] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8564.596376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8570.005715] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8570.219456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8575.974121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.974468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.978829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.986274] LustreError: Skipped 1 previous similar message [ 8576.021892] Lustre: Skipped 3 previous similar messages [ 8578.471876] Lustre: server umount lustre-OST0000 complete [ 8583.146539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8583.165469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8583.180873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8583.190395] Lustre: Skipped 1 previous similar message [ 8586.213427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.138887] Lustre: server umount lustre-MDT0000 complete [ 8591.340401] LustreError: 197299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.365429] LustreError: 197299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8593.553402] LustreError: 197286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391553 with bad export cookie 16518419890281104527 [ 8593.554272] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8593.563757] LustreError: 197286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8594.304201] Lustre: server umount lustre-MDT0001 complete [ 8607.804746] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:39:27 (1787391567) [ 8629.227456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8642.216359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.716277] LustreError: 202394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.796781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.629501] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8652.284975] LustreError: 202395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.315284] LustreError: 202395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8660.943834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.094669] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8674.291341] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8684.000169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8684.508940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.512759] Lustre: Skipped 1 previous similar message [ 8692.204493] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8702.725386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.029806] Lustre: lustre-OST0001: new disk, initializing [ 8703.033411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8703.041689] Lustre: 204702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8709.989629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8710.010569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8710.329296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8714.302831] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8729.778727] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8738.514447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8741.790291] Lustre: DEBUG MARKER: ADD OST3 [ 8748.199817] loop: module loaded [ 8748.291495] loop0: detected capacity change from 0 to 204800000 [ 8748.533256] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8748.569961] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8748.603676] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8756.014983] Lustre: DEBUG MARKER: STOP OST3 [ 8756.304236] loop0: detected capacity change from 0 to 204800000 [ 8763.288278] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8763.594568] Lustre: lustre-OST0003: new disk, initializing [ 8763.599671] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8763.612520] Lustre: 206773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8763.816862] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8763.826916] Lustre: Skipped 1 previous similar message [ 8765.138896] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8765.359618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8765.378506] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8765.437446] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8766.631495] Lustre: Failing over lustre-OST0003 [ 8766.979830] Lustre: server umount lustre-OST0003 complete [ 8767.200670] LustreError: 206862:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8770.554022] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.582513] LustreError: 205748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.585829] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8770.601160] LustreError: 205748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8773.925576] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8789.701876] Lustre: DEBUG MARKER: START OST3 again [ 8799.918230] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.396054] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8801.467760] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8801.647357] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8801.653164] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8808.201939] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8831.444279] Lustre: Failing over lustre-OST0003 [ 8831.485498] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.496201] Lustre: Skipped 1 previous similar message [ 8831.505605] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 8831.517529] Lustre: Skipped 3 previous similar messages [ 8831.600488] Lustre: server umount lustre-OST0003 complete [ 8836.584598] LustreError: 204922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.630231] LustreError: 204922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 8856.555941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8856.562105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.619790] Lustre: Skipped 1 previous similar message [ 8856.646193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.785693] Lustre: server umount lustre-MDT0000 complete [ 8865.410727] LustreError: 202375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391825 with bad export cookie 16518419890281105409 [ 8865.413189] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8865.417151] LustreError: 202375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8872.085482] Lustre: server umount lustre-MDT0001 complete [ 8875.639440] Lustre: server umount lustre-OST0000 complete [ 8879.513963] Lustre: server umount lustre-OST0001 complete [ 8886.228760] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8895.437269] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8935.555661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.830476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8935.862078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8936.003512] Lustre: lustre-MDT0000: new disk, initializing [ 8936.131757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.137284] Lustre: Skipped 1 previous similar message [ 8936.154668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8940.998260] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8950.943428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.073249] Lustre: 212323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8951.301389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8955.758200] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8966.200178] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8973.018724] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8981.557174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.959757] Lustre: lustre-OST0000: new disk, initializing [ 8981.962494] Lustre: Skipped 1 previous similar message [ 8981.970201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8981.974464] Lustre: Skipped 2 previous similar messages [ 8981.981302] Lustre: 214290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8983.379046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8983.399167] Lustre: Skipped 1 previous similar message [ 8983.405047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8983.573949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8990.226945] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9001.835936] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9007.747392] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9008.036893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9014.273727] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9014.613819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9019.362858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.364234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.380307] LustreError: Skipped 1 previous similar message [ 9019.386705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.409544] Lustre: Skipped 6 previous similar messages [ 9019.456096] Lustre: Skipped 9 previous similar messages [ 9023.387649] Lustre: server umount lustre-OST0000 complete [ 9028.067302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9042.400319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9043.024606] Lustre: server umount lustre-MDT0000 complete [ 9044.972277] LustreError: 212333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.995425] LustreError: 212333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9048.704027] LustreError: 212313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392008 with bad export cookie 16518419890281107705 [ 9048.707343] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9048.735637] LustreError: 212313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9049.447641] Lustre: server umount lustre-MDT0001 complete [ 9053.810253] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:46:53 (1787392013) [ 9074.822205] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9085.954835] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9125.577218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.883984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9125.915941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9125.987378] Lustre: lustre-MDT0000: new disk, initializing [ 9126.069740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9130.796383] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9141.849259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.073068] Lustre: 218908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9142.552167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9147.566635] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9159.309657] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9167.356554] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9175.612799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9175.818797] Lustre: 220876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9177.731437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9177.895789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9182.503850] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9194.036721] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9199.984896] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9200.391210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9206.815970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9207.203169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9213.930504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9213.932745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.932754] Lustre: Skipped 2 previous similar messages [ 9213.934167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9213.954164] LustreError: Skipped 1 previous similar message [ 9213.987521] Lustre: Skipped 8 previous similar messages [ 9224.160167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9224.428240] Lustre: server umount lustre-OST0000 complete [ 9235.546637] Lustre: server umount lustre-MDT0000 complete [ 9239.523359] LustreError: 221563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.543853] LustreError: 221563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9240.356394] LustreError: 218900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392200 with bad export cookie 16518419890281108587 [ 9240.366329] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.371442] LustreError: 218900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9240.783371] Lustre: server umount lustre-MDT0001 complete [ 9251.298864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.306105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.308863] Lustre: Skipped 5 previous similar messages [ 9257.744174] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9266.363125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.555207] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9279.754652] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9286.301099] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9293.820761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9300.483330] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9307.140415] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9317.859602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9317.873329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9317.883280] Lustre: Skipped 8 previous similar messages [ 9323.744531] Lustre: server umount lustre-OST0000 complete [ 9334.800062] Lustre: server umount lustre-MDT0000 complete [ 9339.739472] LustreError: 223078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392299 with bad export cookie 16518419890281109469 [ 9339.744964] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9339.754865] LustreError: 223078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9340.244577] Lustre: server umount lustre-MDT0001 complete [ 9448.785745] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9451.883732] Key type lgssc unregistered [ 9452.337937] LNet: 226760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9452.348502] LNetError: 226760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9452.365932] LNet: Removed LNI 192.168.202.110@tcp [ 9453.498222] Key type .llcrypt unregistered [ 9453.501639] Key type ._llcrypt unregistered [ 9484.193006] Key type ._llcrypt registered [ 9484.195299] Key type .llcrypt registered [ 9484.369563] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9500.514708] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9501.404371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9501.480513] alg: No test for adler32 (adler32-zlib) [ 9502.661816] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9502.931402] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9504.632480] Key type lgssc registered [ 9506.039805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9545.721450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9545.779589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.118851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9547.153766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9547.255521] Lustre: lustre-MDT0000: new disk, initializing [ 9547.366930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9547.387587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9551.783398] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9562.233036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.386565] Lustre: 230117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9562.437361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9562.440242] Lustre: Skipped 1 previous similar message [ 9562.493795] Lustre: lustre-MDT0001: new disk, initializing [ 9562.585639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.625724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9562.639190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9567.400782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9577.474830] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9585.426609] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9593.202943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.459154] Lustre: lustre-OST0000: new disk, initializing [ 9593.463537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9593.467744] Lustre: 232084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9593.571491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9599.046038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9599.058839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9599.191853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9601.329244] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9611.651058] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9617.225691] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9617.428301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9622.637852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9622.884901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9630.182658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9630.187581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9630.201876] LustreError: Skipped 1 previous similar message [ 9630.203752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9630.229611] Lustre: Skipped 1 previous similar message [ 9631.449770] Lustre: server umount lustre-OST0000 complete [ 9639.395528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9639.408994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.428760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9639.438119] Lustre: Skipped 1 previous similar message [ 9640.421248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.445406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9640.451091] Lustre: Skipped 1 previous similar message [ 9641.985744] Lustre: server umount lustre-MDT0000 complete [ 9645.726147] LustreError: 230110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392605 with bad export cookie 13340311707168686329 [ 9645.738802] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9646.062134] Lustre: server umount lustre-MDT0001 complete [ 9668.531827] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9679.900331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9680.231697] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.254246] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9680.285652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.478679] LustreError: 235039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.480232] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9690.597038] LustreError: 235038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.716176] LustreError: 235039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.389394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.938052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9701.766561] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9705.422111] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9714.385054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.791785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9722.608318] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9734.958773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.370833] Lustre: lustre-OST0001: new disk, initializing [ 9735.375517] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9735.380965] Lustre: 237345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9735.571192] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.046815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9743.062290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9743.261433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9744.200733] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9757.049033] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9767.956806] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9783.838704] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:59:03 (1787392743) [ 9788.901924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9788.910130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.925410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9792.998392] Lustre: server umount lustre-MDT0000 complete [ 9794.531657] LustreError: 235033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.549018] LustreError: 235033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9797.942396] LustreError: 235018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392757 with bad export cookie 13340311707168687211 [ 9797.943511] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9797.953604] LustreError: 235018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9798.386325] Lustre: server umount lustre-MDT0001 complete [ 9812.568141] Lustre: server umount lustre-OST0000 complete [ 9815.200227] Lustre: 228340:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392759/real 1787392759] req@ffff9c1abdf90a80 x1874216838355200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787392775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9815.237460] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.264632] Lustre: Skipped 3 previous similar messages [ 9817.666249] Lustre: server umount lustre-OST0001 complete [ 9826.969744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9843.548947] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9890.011681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.252711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9890.277931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9890.358217] Lustre: lustre-MDT0000: new disk, initializing [ 9890.474213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9890.490231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9896.620608] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9910.628870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.796994] Lustre: 241320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9910.910146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9910.914671] Lustre: Skipped 1 previous similar message [ 9911.066742] Lustre: lustre-MDT0001: new disk, initializing [ 9911.210442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9911.227439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9916.391801] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9927.545314] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9934.479479] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9942.023701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9942.500331] Lustre: lustre-OST0000: new disk, initializing [ 9942.513522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9942.531835] Lustre: 243299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9943.687070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9943.703776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9943.901293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9951.130658] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9961.640361] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9967.032973] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9967.276147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9972.516782] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9972.823866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9979.879957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9979.882305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.884565] LustreError: Skipped 1 previous similar message [ 9979.886234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9979.893228] Lustre: Skipped 1 previous similar message [ 9979.903725] Lustre: Skipped 5 previous similar messages [ 9989.088183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9989.315543] Lustre: server umount lustre-OST0000 complete [ 9995.239176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.253688] Lustre: Skipped 3 previous similar messages [10000.141729] Lustre: server umount lustre-MDT0000 complete [10000.353908] LustreError: 241325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.391723] LustreError: 241325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10004.506836] LustreError: 241310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392964 with bad export cookie 13340311707168689346 [10004.510298] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10004.516812] LustreError: 241310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10004.948321] Lustre: server umount lustre-MDT0001 complete [10015.537898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.308507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.331504] Lustre: Skipped 2 previous similar messages [10021.356599] LustreError: 245517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.388322] LustreError: 245517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10021.772703] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10032.742535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.264575] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10049.753212] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10057.355817] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10066.774616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10074.968378] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10084.292767] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10232.292253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10232.312744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.323158] Lustre: Skipped 2 previous similar messages [10232.332559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10245.088638] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10245.397722] Lustre: server umount lustre-OST0000 complete [10256.320444] Lustre: server umount lustre-MDT0000 complete [10260.964268] LustreError: 245512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.008753] LustreError: 245512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10261.259086] LustreError: 245499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393221 with bad export cookie 13340311707168690228 [10261.284233] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10261.961871] Lustre: server umount lustre-MDT0001 complete [10355.234445] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10358.234979] Key type lgssc unregistered [10358.649956] LNet: 249126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10358.662545] LNetError: 249126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10359.726849] LNet: Removed LNI 192.168.202.110@tcp [10360.747619] Key type .llcrypt unregistered [10360.755965] Key type ._llcrypt unregistered [10390.711079] Key type ._llcrypt registered [10390.713343] Key type .llcrypt registered [10390.839360] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [10408.888446] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10410.194756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10410.333210] alg: No test for adler32 (adler32-zlib) [10411.505956] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [10411.762442] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10413.520412] Key type lgssc registered [10414.756818] Lustre: Echo OBD driver; http://www.lustre.org/ [10467.232829] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10478.996360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10479.118935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.392676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10480.434891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10480.539317] Lustre: lustre-MDT0000: new disk, initializing [10480.639814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.669669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10486.117094] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10499.239318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.395991] Lustre: 253199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10499.512758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10499.519341] Lustre: Skipped 1 previous similar message [10499.649261] Lustre: lustre-MDT0001: new disk, initializing [10499.818528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10499.915246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10499.934478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10505.188607] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10511.340489] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10519.320360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10519.577501] Lustre: lustre-OST0000: new disk, initializing [10519.581421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10519.587532] Lustre: 254830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10519.690613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10524.258698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10524.276077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10524.432438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10527.678845] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10541.105918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10541.353511] Lustre: lustre-OST0001: new disk, initializing [10541.360688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10541.372120] Lustre: 255701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10541.564052] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10550.389570] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10550.504375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10550.520839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10550.692704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10563.199881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10575.114966] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10591.733309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.744758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.784351] Lustre: Skipped 2 previous similar messages [10591.835579] Lustre: Skipped 3 previous similar messages [10592.619436] Lustre: server umount lustre-MDT0000 complete [10598.565978] LustreError: 256472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393558 with bad export cookie 18054364578778374341 [10598.568400] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10598.584223] LustreError: 256472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10599.363885] Lustre: server umount lustre-MDT0001 complete [10614.428108] Lustre: server umount lustre-OST0000 complete [10618.144233] Lustre: 250705:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393562/real 1787393562] req@ffff9c1aa4cf5180 x1874217791471616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10618.180152] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.195749] Lustre: Skipped 1 previous similar message [10619.533692] Lustre: server umount lustre-OST0001 complete [10635.749764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10647.442590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10657.987119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10670.325995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10693.820615] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10707.072514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.099590] Lustre: lustre-MDT0000: reset Object Index mappings [10707.880748] LustreError: 260718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10707.919264] LustreError: 260718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10708.013706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10713.065228] LustreError: 260719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.480499] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10717.176089] LustreError: 260718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.283450] LustreError: 260719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.322281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.340511] Lustre: lustre-MDT0001: reset Object Index mappings [10725.918527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10732.197471] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10736.467798] Lustre: 261862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10743.819705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.254913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10750.469143] LustreError: 262217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10751.290929] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10752.507046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10760.428568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.851315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10767.665533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10776.170814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10780.347639] Lustre: 263729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10786.274717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10786.286533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10786.296437] Lustre: Skipped 2 previous similar messages [10786.323548] Lustre: Skipped 3 previous similar messages [10788.107906] Lustre: server umount lustre-MDT0000 complete [10791.400869] LustreError: 260718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.439319] LustreError: 260718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10792.728571] LustreError: 260700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393752 with bad export cookie 18054364578778381131 [10792.734538] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10792.746623] LustreError: 260700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10793.235950] Lustre: server umount lustre-MDT0001 complete [10808.653942] Lustre: server umount lustre-OST0000 complete [10812.896146] Lustre: 250703:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393756/real 1787393756] req@ffff9c1a98c9f800 x1874217791523840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787393772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10812.938844] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10812.965433] Lustre: Skipped 1 previous similar message [10813.385593] Lustre: server umount lustre-OST0001 complete [10839.135536] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:16:38 (1787393798) [10859.701373] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [10872.388311] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10910.128506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.376670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10910.404347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10910.533221] Lustre: lustre-MDT0000: new disk, initializing [10910.635676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10910.644430] Lustre: Skipped 1 previous similar message [10910.656980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10915.685176] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10926.808520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.912818] Lustre: 267989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10926.976489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10926.985080] Lustre: Skipped 1 previous similar message [10927.072524] Lustre: lustre-MDT0001: new disk, initializing [10927.285431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10927.298961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10931.984514] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10941.897523] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10948.248456] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10956.411284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10956.703450] Lustre: lustre-OST0000: new disk, initializing [10956.706766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10956.712557] Lustre: 269960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10958.557764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10958.582563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10958.706258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10963.784765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10973.446509] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10978.578335] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10978.818560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10984.514508] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10984.781157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10989.542180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10989.543785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.584399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10994.657801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10994.667157] Lustre: Skipped 2 previous similar messages [10999.783333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10999.793639] Lustre: Skipped 1 previous similar message [11001.312143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11001.501613] Lustre: server umount lustre-OST0000 complete [11008.995944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11008.998676] LustreError: Skipped 1 previous similar message [11009.017774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11009.037914] Lustre: Skipped 1 previous similar message [11009.044974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11020.256148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11020.473833] Lustre: server umount lustre-MDT0000 complete [11024.240818] LustreError: 267980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393984 with bad export cookie 18054364578778383224 [11024.251329] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11024.251571] LustreError: 267980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11024.762935] Lustre: server umount lustre-MDT0001 complete [11045.584343] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11057.639053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.098523] LustreError: 272896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.158448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11058.167650] Lustre: Skipped 2 previous similar messages [11063.842737] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11074.675243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.924707] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11083.239761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11091.338568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.649955] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11107.713552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.894748] Lustre: lustre-OST0001: new disk, initializing [11107.899708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11107.907413] Lustre: 275206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11109.374902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11109.388974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11109.578404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11116.416408] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11126.635548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11130.601589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11137.114829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11137.172864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11145.700455] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.711783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11145.717155] Lustre: Skipped 3 previous similar messages [11145.743302] Lustre: Skipped 9 previous similar messages [11150.821911] LustreError: 272893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.889491] LustreError: 272893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11151.025394] Lustre: server umount lustre-MDT0000 complete [11156.248851] LustreError: 272878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394116 with bad export cookie 18054364578778384106 [11156.250601] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11156.270503] LustreError: 272878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11157.225594] Lustre: server umount lustre-MDT0001 complete [11172.788020] Lustre: server umount lustre-OST0000 complete [11187.070973] Lustre: server umount lustre-OST0001 complete [11205.419392] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11217.830769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11218.210709] LustreError: 278263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.224953] LustreError: 278263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11218.277510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11218.281439] Lustre: Skipped 3 previous similar messages [11223.087201] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11232.443282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.362938] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11241.718675] Lustre: 279403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11249.075710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11257.247418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11264.029023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11270.096850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11275.830166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11278.848552] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11287.719851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11292.077106] Lustre: 281281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11298.523163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11298.740602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11310.444429] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:24:29 (1787394269) [11326.955738] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.959758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.982783] Lustre: Skipped 4 previous similar messages [11327.029994] Lustre: Skipped 3 previous similar messages [11333.179215] Lustre: server umount lustre-MDT0000 complete [11338.116638] LustreError: 278245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394298 with bad export cookie 18054364578778390637 [11338.130272] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11344.424888] Lustre: server umount lustre-MDT0001 complete [11363.296343] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11363.432733] Lustre: server umount lustre-OST0000 complete [11368.487259] Lustre: server umount lustre-OST0001 complete [11379.579248] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [11396.470978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11449.813951] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11461.880034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.121715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11462.175745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11462.291281] Lustre: lustre-MDT0000: new disk, initializing [11462.479439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11467.679866] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11480.184443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.345961] Lustre: 285543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11480.551557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11480.555650] Lustre: Skipped 4 previous similar messages [11480.600537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.460501] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11497.493828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.874631] Lustre: 287178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11500.064750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11504.134177] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11515.847320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11516.104658] Lustre: 288050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11517.512863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11517.531881] Lustre: Skipped 1 previous similar message [11517.786416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11524.358630] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11534.963461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11538.891504] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11538.894838] Lustre: Skipped 1 previous similar message [11556.722511] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:28:36 (1787394516) [11562.082152] Lustre: server umount lustre-MDT0000 complete [11562.469755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11562.476909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.497951] Lustre: Skipped 2 previous similar messages [11562.523046] LustreError: 285554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11562.554198] LustreError: 285554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [11567.138817] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394527 with bad export cookie 18054364578778398274 [11567.142922] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11567.154385] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11567.567850] Lustre: server umount lustre-MDT0001 complete [11581.287537] Lustre: server umount lustre-OST0000 complete [11595.934069] Lustre: server umount lustre-OST0001 complete [11607.224052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.586823] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11623.836471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.342688] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11637.526961] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11644.585705] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11653.459221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11662.199463] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11671.210331] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11700.828290] loop1: detected capacity change from 0 to 204800000 [11701.152277] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11701.190266] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11701.201839] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11706.769392] loop1: detected capacity change from 0 to 204800000 [11713.068043] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.176992] LustreError: 290283:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9929: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11713.193231] LustreError: 290283:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9929 [11713.199121] LustreError: 290283:0:(mgs_handler.c:517:mgs_target_reg()) fs9929-MDT0001: Error checking target index: rc = -13 [11713.206734] LustreError: fs9929-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11713.214316] LustreError: 294038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11713.218762] LustreError: 294038:0:(tgt_mount.c:2129:server_put_super()) no obd fs9929-MDT0001 [11713.221068] LustreError: 294038:0:(tgt_mount.c:116:server_deregister_mount()) fs9929-MDT0001 not registered [11713.245868] Lustre: server umount fs9929-MDT0001 complete [11713.249336] LustreError: 294038:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11721.557228] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.662295] Lustre: Setting parameter fs9929-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9929-MDT0001 [11721.709864] Lustre: srv-fs9929-MDT0001: No data found on store. Initialize space. [11721.713757] Lustre: Skipped 4 previous similar messages [11721.785756] Lustre: fs9929-MDT0001: new disk, initializing [11721.789139] Lustre: Skipped 3 previous similar messages [11721.903603] Lustre: srv-fs9929-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11721.911789] Lustre: 294504:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9929-MDT0001-mdtlov: can't create new llog: rc = -115 [11721.922651] LustreError: 294504:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11726.926522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11735.449228] Lustre: server umount fs9929-MDT0001 complete [11749.364254] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11750.541362] loop1: detected capacity change from 0 to 204800000 [11754.482744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11758.589019] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11759.529354] Lustre: 296073:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9929-MDT0001-mdtlov: can't create new llog: rc = -115 [11759.548232] LustreError: 296073:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11767.095804] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11772.606995] Lustre: server umount fs9929-MDT0001 complete [11784.894115] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [11784.903240] Lustre: Skipped 8 previous similar messages [11785.186247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.199206] Lustre: Skipped 5 previous similar messages [11790.386271] Lustre: server umount lustre-MDT0000 complete [11799.910414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.136260] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.687582] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11805.693351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11814.951972] loop1: detected capacity change from 0 to 204800000 [11820.729230] LustreError: 292125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.738353] LustreError: 292125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 98 previous similar messages [11822.527500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.273288] Lustre: 298520:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9929-MDT0001-mdtlov: can't create new llog: rc = -115 [11823.289720] LustreError: 298520:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9929-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11829.717735] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11835.806139] Lustre: server umount fs9929-MDT0001 complete [11861.990549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11876.320189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11876.461669] Lustre: server umount lustre-OST0000 complete [11882.465471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11886.513313] Lustre: server umount lustre-MDT0000 complete [11890.259203] LustreError: 290276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394850 with bad export cookie 18054364578778400955 [11890.267230] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11890.272682] LustreError: 290276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11890.638849] Lustre: server umount lustre-MDT0001 complete [11901.316319] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11903.936834] Key type lgssc unregistered [11904.279680] LNet: 300866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11904.297107] LNetError: 300866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11904.332248] LNet: Removed LNI 192.168.202.110@tcp [11905.352147] Key type .llcrypt unregistered [11905.354552] Key type ._llcrypt unregistered [11927.028888] Key type ._llcrypt registered [11927.030546] Key type .llcrypt registered [11927.100725] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11928.288914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11928.347045] alg: No test for adler32 (adler32-zlib) [11929.349335] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11929.551479] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11931.176207] Key type lgssc registered [11932.340765] Lustre: Echo OBD driver; http://www.lustre.org/ [11940.970884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11940.994290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11942.696362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11947.629320] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11952.114190] LustreError: 302142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.136623] LustreError: 302142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11957.217171] LustreError: 302141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.829917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.268612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11963.647448] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11972.096797] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11979.230216] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11988.248566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11988.628490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11996.919709] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12002.994641] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.083880] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12008.432576] LustreError: 303971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.536567] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.599615] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12022.769306] LustreError: 303970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.807418] LustreError: 303970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12033.016398] loop1: detected capacity change from 0 to 204800000 [12033.300183] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12033.321310] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12033.341852] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12039.328186] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12039.346904] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [12039.956911] loop1: detected capacity change from 0 to 204800000 [12047.037030] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12047.271588] Lustre: fs22020-OST0000: new disk, initializing [12047.274930] Lustre: srv-fs22020-OST0000: No data found on store. Initialize space. [12047.280752] Lustre: 305880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12047.401120] Lustre: fs22020-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12055.393855] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12064.746716] Lustre: server umount fs22020-OST0000 complete [12073.957160] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.995360] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [12077.818293] loop1: detected capacity change from 0 to 204800000 [12084.214255] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12084.958668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12085.187657] Lustre: fs22020-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12089.338840] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12092.354531] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12104.020147] loop2: detected capacity change from 0 to 204800000 [12104.313520] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12104.332403] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12104.348330] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12111.608855] loop2: detected capacity change from 0 to 204800000 [12118.798394] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.962353] LustreError: fs22020-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12118.978358] LustreError: fs22020-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12118.988727] LustreError: 308920:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12118.995460] LustreError: 308920:0:(tgt_mount.c:2129:server_put_super()) no obd fs22020-OST0001 [12119.013983] LustreError: 308920:0:(tgt_mount.c:116:server_deregister_mount()) fs22020-OST0001 not registered [12119.060879] Lustre: server umount fs22020-OST0001 complete [12119.066985] LustreError: 308920:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12131.115833] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12131.223973] Lustre: fs22020-OST0001: new disk, initializing [12131.227859] Lustre: srv-fs22020-OST0001: No data found on store. Initialize space. [12131.231837] Lustre: 309471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12131.299158] Lustre: fs22020-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12138.202506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12140.527882] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.542985] LustreError: 303968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [12147.943966] Lustre: server umount fs22020-OST0000 complete [12158.269520] Lustre: server umount fs22020-OST0001 complete [12173.606993] loop1: detected capacity change from 0 to 204800000 [12179.752726] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12179.989464] Lustre: fs22020-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12186.431339] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12193.894656] loop2: detected capacity change from 0 to 204800000 [12200.818209] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12209.287709] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12221.508450] Lustre: server umount fs22020-OST0000 complete [12232.447288] Lustre: server umount fs22020-OST0001 complete [12237.803847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12242.921607] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12248.036619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12248.054249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.070835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.080605] Lustre: Skipped 1 previous similar message [12248.115970] Lustre: Skipped 1 previous similar message [12251.925694] Lustre: server umount lustre-OST0000 complete [12258.273634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.287409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12258.307255] Lustre: Skipped 1 previous similar message [12258.327239] Lustre: Skipped 1 previous similar message [12262.892239] Lustre: server umount lustre-MDT0000 complete [12267.379272] LustreError: 302122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395227 with bad export cookie 8880896966174545415 [12267.384059] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12267.397805] LustreError: 302122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12267.916610] Lustre: server umount lustre-MDT0001 complete [12283.107705] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12286.470406] Key type lgssc unregistered [12286.797156] LNet: 315398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12286.818686] LNetError: 315398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12286.842348] LNet: Removed LNI 192.168.202.110@tcp [12287.878160] Key type .llcrypt unregistered [12287.883376] Key type ._llcrypt unregistered [12328.405351] Key type ._llcrypt registered [12328.407712] Key type .llcrypt registered [12328.528068] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [12349.797909] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12351.450163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.786261] alg: No test for adler32 (adler32-zlib) [12353.070550] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12353.383645] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12355.272206] Key type lgssc registered [12357.231123] Lustre: Echo OBD driver; http://www.lustre.org/ [12398.366357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12398.411784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12399.759895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12399.820667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12399.916678] Lustre: lustre-MDT0000: new disk, initializing [12400.021687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12400.065645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12404.659259] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12416.394266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12416.520943] Lustre: 319268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12416.606652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12416.614423] Lustre: Skipped 1 previous similar message [12416.710721] Lustre: lustre-MDT0001: new disk, initializing [12416.885845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12416.955952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12416.973677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12422.020638] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12432.675043] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12439.982783] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12447.977436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12448.268510] Lustre: lustre-OST0000: new disk, initializing [12448.275211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12448.290862] Lustre: 321239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12448.451824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12455.519650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12455.535346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12455.649140] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12455.723386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12467.160307] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12470.958127] Lustre: Mounted lustre-client [12472.270092] Lustre: Unmounted lustre-client [12486.630932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12486.631213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.640236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12486.669392] Lustre: Skipped 1 previous similar message [12490.046104] Lustre: server umount lustre-OST0000 complete [12496.868787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.883628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.885558] Lustre: Skipped 1 previous similar message [12501.038201] Lustre: server umount lustre-MDT0000 complete [12504.994854] LustreError: 321238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395465 with bad export cookie 12073304405788568159 [12504.996061] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12505.011710] LustreError: 321238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12505.521869] Lustre: server umount lustre-MDT0001 complete [12517.927448] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12520.834696] Key type lgssc unregistered [12522.464337] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [12523.232640] LNet: 323398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12523.245857] LNetError: 323398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12601.063410] LNet: Removed LNI 192.168.202.110@tcp [12602.169185] Key type .llcrypt unregistered [12602.170954] Key type ._llcrypt unregistered [12621.091913] Key type ._llcrypt registered [12621.097856] Key type .llcrypt registered [12621.193664] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [12639.744259] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12641.005578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12641.080967] alg: No test for adler32 (adler32-zlib) [12642.194191] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12642.412621] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12644.064213] Key type lgssc registered [12645.181290] Lustre: Echo OBD driver; http://www.lustre.org/ [12678.312734] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:47:18 (1787395638) [12684.845872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12684.870374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.077668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12686.103443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12686.170284] Lustre: lustre-MDT0000: new disk, initializing [12686.285868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12686.312094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12691.289449] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12702.516551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12702.741073] Lustre: 326628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12702.843178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12702.848606] Lustre: Skipped 1 previous similar message [12702.958881] Lustre: lustre-MDT0001: new disk, initializing [12703.152728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.215115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12703.229275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12707.668239] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12718.556070] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12726.404463] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12736.839315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12737.343637] Lustre: lustre-OST0000: new disk, initializing [12737.349379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12737.370851] Lustre: 328595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12737.489304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12742.728111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12742.757122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12742.850429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12744.948684] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12754.343493] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12773.868907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12773.875225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12773.903826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12773.914361] Lustre: Skipped 1 previous similar message [12776.974907] Lustre: server umount lustre-OST0000 complete [12784.100387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.110642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12784.116258] Lustre: Skipped 2 previous similar messages [12784.135179] Lustre: Skipped 1 previous similar message [12786.721943] Lustre: server umount lustre-MDT0000 complete [12790.488251] LustreError: 327422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395750 with bad export cookie 2677038176174185045 [12790.501772] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12791.228337] Lustre: server umount lustre-MDT0001 complete [12802.726149] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12805.637878] Key type lgssc unregistered [12805.915917] LNet: 330501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12805.929347] LNetError: 330501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12806.956286] LNet: Removed LNI 192.168.202.110@tcp [12808.010579] Key type .llcrypt unregistered [12808.013950] Key type ._llcrypt unregistered [12832.692752] Key type ._llcrypt registered [12832.695433] Key type .llcrypt registered [12832.835366] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12833.833220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12833.851982] alg: No test for adler32 (adler32-zlib) [12835.132369] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12835.363949] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12837.083064] Key type lgssc registered [12838.534768] Lustre: Echo OBD driver; http://www.lustre.org/ [12847.004746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12847.042300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12848.541996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12853.567805] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12858.858173] LustreError: 331774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.878170] LustreError: 331774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12861.991322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12862.266631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12866.790980] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12874.627351] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12881.052914] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12888.739872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.213802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12896.524564] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12904.333465] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12917.857547] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12919.823188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12926.666315] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [12928.993526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12929.014318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12929.050073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12931.020759] Lustre: server umount lustre-MDT0000 complete [12931.745076] LustreError: 334265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.650106] LustreError: 331775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.663521] LustreError: 331775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12935.644169] LustreError: 331753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395895 with bad export cookie 12990862508697709067 [12935.682613] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12936.861882] LustreError: 331770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12936.903964] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [12936.920020] Lustre: Skipped 3 previous similar messages [12939.756263] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12939.774626] Lustre: Skipped 2 previous similar messages [12939.785687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12941.990703] LustreError: 331770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.023190] LustreError: 331770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12942.439879] Lustre: server umount lustre-MDT0001 complete [12947.076429] Lustre: server umount lustre-OST0000 complete [12958.466364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.395772] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12968.933232] Lustre: 335956:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.110@tcp [12971.747540] Lustre: server umount MGS complete [12981.444874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12982.282912] LustreError: 336529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12982.402312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.915341] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12997.403289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12997.814493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13002.983425] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13013.112996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13013.488179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13019.723629] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13028.159848] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13030.275046] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13037.323357] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13039.534596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13045.739569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13045.744118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13045.765774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13045.769061] Lustre: Skipped 1 previous similar message [13046.884530] Lustre: server umount lustre-OST0000 complete [13053.410418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13053.446728] Lustre: Skipped 1 previous similar message [13058.468643] Lustre: server umount lustre-MDT0000 complete [13058.534306] LustreError: 337726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.582828] LustreError: 337726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13062.588204] LustreError: 336509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396022 with bad export cookie 12990862508697709907 [13062.600616] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13062.609383] LustreError: 336509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13073.195860] Lustre: server umount lustre-MDT0001 complete [13090.655332] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13093.450487] Key type lgssc unregistered [13093.817401] LNet: 339988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13093.824509] LNetError: 339988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13093.850854] LNet: Removed LNI 192.168.202.110@tcp [13095.124806] Key type .llcrypt unregistered [13095.131966] Key type ._llcrypt unregistered [13119.577590] Key type ._llcrypt registered [13119.579929] Key type .llcrypt registered [13119.765751] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [13141.778810] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13143.133506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13143.166991] alg: No test for adler32 (adler32-zlib) [13144.431686] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [13144.740433] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13146.456172] Key type lgssc registered [13147.793620] Lustre: Echo OBD driver; http://www.lustre.org/ [13187.654882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13187.711537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13189.121981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13189.158757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13189.280582] Lustre: lustre-MDT0000: new disk, initializing [13189.373417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13189.395572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13194.213831] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13205.842546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13206.012767] Lustre: 343150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13206.096749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13206.102414] Lustre: Skipped 1 previous similar message [13206.225863] Lustre: lustre-MDT0001: new disk, initializing [13206.355123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13206.401323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13206.421327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13212.031819] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13222.452766] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13229.774422] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13237.664350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13237.947614] Lustre: lustre-OST0000: new disk, initializing [13237.951684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13237.957592] Lustre: 345122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13238.081149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13244.063301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13244.080146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13244.274652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13245.067255] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13255.756320] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13261.058732] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13261.229777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13266.663772] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13266.995878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13269.985511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13269.991159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13269.999782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13270.016367] Lustre: Skipped 1 previous similar message [13270.037960] Lustre: Skipped 1 previous similar message [13275.108898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13275.121881] Lustre: Skipped 1 previous similar message [13280.231813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13280.240400] Lustre: Skipped 1 previous similar message [13283.298946] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13283.630503] Lustre: server umount lustre-OST0000 complete [13288.418638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13288.431744] LustreError: Skipped 1 previous similar message [13288.437934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13288.455545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13290.469886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.587140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13295.594726] Lustre: Skipped 3 previous similar messages [13302.240202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13302.476670] Lustre: server umount lustre-MDT0000 complete [13308.030193] LustreError: 343145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396268 with bad export cookie 11731987931346806673 [13308.034571] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13308.048466] LustreError: 343145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13308.552977] Lustre: server umount lustre-MDT0001 complete [13319.168248] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:57:58 (1787396278) [13338.431877] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [13350.601417] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13390.348669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13390.635722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13390.666301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13390.730605] Lustre: lustre-MDT0000: new disk, initializing [13390.790797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13390.805169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13395.775575] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13406.790466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.983212] Lustre: 349889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13407.116643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13407.127284] Lustre: Skipped 1 previous similar message [13407.285558] Lustre: lustre-MDT0001: new disk, initializing [13407.450653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13407.522044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13407.529056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13412.118615] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13425.790339] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13433.815235] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13441.373234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13441.639439] Lustre: lustre-OST0000: new disk, initializing [13441.642371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13441.659069] Lustre: 351860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13441.803496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13450.015395] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13451.316081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13451.327432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13451.462187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13461.554463] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13508.066749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13508.078475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13508.112617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13508.145254] Lustre: Skipped 3 previous similar messages [13512.847580] Lustre: server umount lustre-OST0000 complete [13518.324772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13518.329515] Lustre: Skipped 2 previous similar messages [13523.938396] Lustre: server umount lustre-MDT0000 complete [13528.554365] LustreError: 352679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.575876] LustreError: 352679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13528.903690] LustreError: 349880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396488 with bad export cookie 11731987931346807555 [13528.921972] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13528.940412] LustreError: 349880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13529.444745] Lustre: server umount lustre-MDT0001 complete [13544.094328] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13547.862805] Key type lgssc unregistered [13548.420180] LNet: 354048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13548.441685] LNetError: 354048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13549.478419] LNet: Removed LNI 192.168.202.110@tcp [13550.467163] Key type .llcrypt unregistered [13550.468963] Key type ._llcrypt unregistered [13672.556850] Key type ._llcrypt registered [13672.566206] Key type .llcrypt registered [13672.744179] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13673.132615] Key type .llcrypt unregistered [13673.139545] Key type ._llcrypt unregistered [13766.019952] Key type ._llcrypt registered [13766.023922] Key type .llcrypt registered [13766.215886] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13766.613436] Key type .llcrypt unregistered [13766.615243] Key type ._llcrypt unregistered [13805.088350] Key type ._llcrypt registered [13805.091964] Key type .llcrypt registered [13805.233349] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13805.573774] Key type .llcrypt unregistered [13805.577584] Key type ._llcrypt unregistered [13841.577134] Key type ._llcrypt registered [13841.579705] Key type .llcrypt registered [13841.707708] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13842.082274] Key type .llcrypt unregistered [13842.089158] Key type ._llcrypt unregistered [13934.338798] Key type ._llcrypt registered [13934.354285] Key type .llcrypt registered [13934.479310] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13934.821391] Key type .llcrypt unregistered [13934.828692] Key type ._llcrypt unregistered [13975.487114] Key type ._llcrypt registered [13975.493454] Key type .llcrypt registered [13975.738553] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13976.117675] Key type .llcrypt unregistered [13976.123096] Key type ._llcrypt unregistered [14014.413871] Key type ._llcrypt registered [14014.418950] Key type .llcrypt registered [14014.621918] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14014.960611] Key type .llcrypt unregistered [14014.964634] Key type ._llcrypt unregistered [14053.425196] Key type ._llcrypt registered [14053.426924] Key type .llcrypt registered [14053.578819] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14053.905464] Key type .llcrypt unregistered [14053.907745] Key type ._llcrypt unregistered [14084.686915] Key type ._llcrypt registered [14084.691570] Key type .llcrypt registered [14084.857850] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14085.097429] Key type .llcrypt unregistered [14085.099741] Key type ._llcrypt unregistered [14121.573572] Key type ._llcrypt registered [14121.576655] Key type .llcrypt registered [14121.695906] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14122.021234] Key type .llcrypt unregistered [14122.023437] Key type ._llcrypt unregistered [14195.519568] Key type ._llcrypt registered [14195.524801] Key type .llcrypt registered [14195.616466] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14195.898467] Key type .llcrypt unregistered [14195.901545] Key type ._llcrypt unregistered [14236.523414] Key type ._llcrypt registered [14236.525473] Key type .llcrypt registered [14236.612396] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14236.832542] Key type .llcrypt unregistered [14236.834958] Key type ._llcrypt unregistered [14269.450858] Key type ._llcrypt registered [14269.452569] Key type .llcrypt registered [14269.525237] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14269.852403] Key type .llcrypt unregistered [14269.856221] Key type ._llcrypt unregistered [14333.800169] Key type ._llcrypt registered [14333.805041] Key type .llcrypt registered [14333.948718] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14351.697544] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14352.805761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14352.923108] alg: No test for adler32 (adler32-zlib) [14354.033464] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [14354.296110] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14355.976221] Key type lgssc registered [14357.314833] Lustre: Echo OBD driver; http://www.lustre.org/ [14400.189658] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14400.235496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14401.800166] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14401.857206] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14402.015185] Lustre: tst250fs-MDT0000: new disk, initializing [14402.109837] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14402.148391] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14407.659974] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14419.358325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14419.433556] Lustre: 374424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14419.480693] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14419.484157] Lustre: Skipped 1 previous similar message [14419.537627] Lustre: tst250fs-MDT0001: new disk, initializing [14419.585968] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14419.639390] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14419.653450] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14424.616138] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14429.478030] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14436.772794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14437.174693] Lustre: tst250fs-OST0000: new disk, initializing [14437.177861] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14437.185283] Lustre: 376056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14437.270233] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14438.610495] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14438.627227] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14438.771504] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14443.782976] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14455.944533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14456.182392] Lustre: tst250fs-OST0001: new disk, initializing [14456.192351] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14456.205673] Lustre: 376932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14456.398995] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14463.164573] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14463.181347] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14463.327171] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14463.443463] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14469.545898] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14471.700188] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14471.706499] Lustre: Skipped 1 previous similar message [14494.182891] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14494.191789] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14494.210082] Lustre: Skipped 1 previous similar message [14494.234730] Lustre: Skipped 3 previous similar messages [14498.357151] Lustre: server umount tst250fs-MDT0000 complete [14502.702906] LustreError: 374416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397462 with bad export cookie 12327731364118813003 [14502.704817] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14502.712185] LustreError: 374416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14503.165179] Lustre: server umount tst250fs-MDT0001 complete [14518.059470] Lustre: server umount tst250fs-OST0000 complete [14520.800116] Lustre: 372649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397464/real 1787397464] req@ffff9c1a85161f80 x1874221925998976/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14520.835772] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14520.845732] Lustre: Skipped 2 previous similar messages [14522.335519] Lustre: server umount tst250fs-OST0001 complete [14545.704108] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14556.181668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14591.939580] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:19:11 (1787397551) [14594.081470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14611.928175] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14624.138440] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14673.134844] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14424 sec ========= 07:20:32 (1787397632) [14675.655716] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:20:34 (1787397634) === [14679.979586] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:20:39 (1787397639) === [14729.121250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14732.261949] Key type lgssc unregistered [14732.676606] LNet: 386075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14732.694153] LNetError: 386075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14733.746128] LNet: Removed LNI 192.168.202.110@tcp [14735.011294] Key type .llcrypt unregistered [14735.014118] Key type ._llcrypt unregistered