[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 453484391 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003280] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008368] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012038] LSM: Security Framework initializing [ 0.014052] Yama: becoming mindful. [ 0.015050] SELinux: Initializing. [ 0.016081] *** VALIDATE selinux *** [ 0.024039] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029238] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] *** VALIDATE tmpfs *** [ 0.034440] *** VALIDATE proc *** [ 0.036122] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039067] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.046256] debug: unmapping init [mem 0xffffffffbd659000-0xffffffffbd660fff] [ 0.048186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050024] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055016] ... fixed-purpose events: 3 [ 0.056011] ... event mask: 000000070000000f [ 0.057305] rcu: Hierarchical SRCU implementation. [ 0.059434] smp: Bringing up secondary CPUs ... [ 0.060594] x86: Booting SMP configuration: [ 0.061025] .... node #0, CPUs: #1 #2 #3 [ 0.066013] smp: Brought up 1 node, 4 CPUs [ 0.068015] smpboot: Max logical packages: 1 [ 0.069009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151471] node 0 deferred pages initialised in 79ms [ 0.155170] devtmpfs: initialized [ 0.157253] x86/mm: Memory block size: 128MB [ 0.159993] gcov: version magic: 0x41383552 [ 0.163265] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168270] pinctrl core: initialized pinctrl subsystem [ 0.170136] [ 0.170521] ************************************************************* [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174009] ** ** [ 0.176014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179012] ** ** [ 0.181012] ** This means that this kernel is built to expose internal ** [ 0.183014] ** IOMMU data structures, which may compromise security on ** [ 0.186013] ** your system. ** [ 0.188008] ** ** [ 0.191014] ** If you see this message and you are not debugging the ** [ 0.193010] ** kernel, report this immediately to your vendor! ** [ 0.195018] ** ** [ 0.198010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200012] ************************************************************* [ 0.202511] NET: Registered protocol family 16 [ 0.204303] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214046] cpuidle: using governor menu [ 0.216612] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218466] PCI: Using configuration type 1 for base access [ 0.221130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.238142] cryptd: max_cpu_qlen set to 1000 [ 0.241302] ACPI: Added _OSI(Module Device) [ 0.243019] ACPI: Added _OSI(Processor Device) [ 0.245016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252262] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259473] ACPI: Interpreter enabled [ 0.261065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262017] ACPI: Using IOAPIC for interrupt routing [ 0.264120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278830] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292900] acpiphp: Slot [2] registered [ 0.293000] acpiphp: Slot [5] registered [ 0.295183] acpiphp: Slot [6] registered [ 0.297152] acpiphp: Slot [7] registered [ 0.299235] acpiphp: Slot [8] registered [ 0.300131] acpiphp: Slot [9] registered [ 0.302147] acpiphp: Slot [10] registered [ 0.304139] acpiphp: Slot [3] registered [ 0.306113] acpiphp: Slot [4] registered [ 0.307112] acpiphp: Slot [11] registered [ 0.308207] acpiphp: Slot [12] registered [ 0.310108] acpiphp: Slot [13] registered [ 0.311083] acpiphp: Slot [14] registered [ 0.313128] acpiphp: Slot [15] registered [ 0.314127] acpiphp: Slot [16] registered [ 0.316143] acpiphp: Slot [17] registered [ 0.318119] acpiphp: Slot [18] registered [ 0.319106] acpiphp: Slot [19] registered [ 0.321109] acpiphp: Slot [20] registered [ 0.322096] acpiphp: Slot [21] registered [ 0.324113] acpiphp: Slot [22] registered [ 0.326111] acpiphp: Slot [23] registered [ 0.327142] acpiphp: Slot [24] registered [ 0.329093] acpiphp: Slot [25] registered [ 0.331126] acpiphp: Slot [26] registered [ 0.332114] acpiphp: Slot [27] registered [ 0.334173] acpiphp: Slot [28] registered [ 0.335107] acpiphp: Slot [29] registered [ 0.337123] acpiphp: Slot [30] registered [ 0.339101] acpiphp: Slot [31] registered [ 0.340096] PCI host bridge to bus 0000:00 [ 0.342017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361378] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365330] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374781] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380990] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.386020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.391595] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395081] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400775] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425785] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429945] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.443017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469671] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.475014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.529016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.550026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559110] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.598016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.610954] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653680] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.661016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.670015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.687019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.697178] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.701440] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.703356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709624] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714046] iommu: Default domain type: Passthrough [ 0.716393] SCSI subsystem initialized [ 0.719171] ACPI: bus type USB registered [ 0.720118] usbcore: registered new interface driver usbfs [ 0.723096] usbcore: registered new interface driver hub [ 0.725092] usbcore: registered new device driver usb [ 0.728181] pps_core: LinuxPPS API ver. 1 registered [ 0.730017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.734066] PTP clock support registered [ 0.737112] EDAC MC: Ver: 3.0.0 [ 0.738625] PCI: Using ACPI for IRQ routing [ 0.741898] NetLabel: Initializing [ 0.743010] NetLabel: domain hash size = 128 [ 0.745012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.747076] NetLabel: unlabeled traffic allowed by default [ 0.750091] vgaarb: loaded [ 0.751262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.752000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759737] clocksource: Switched to clocksource kvm-clock [ 0.881914] VFS: Disk quotas dquot_6.6.0 [ 0.883762] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886668] *** VALIDATE ramfs *** [ 0.887686] *** VALIDATE hugetlbfs *** [ 0.889418] pnp: PnP ACPI init [ 0.891979] pnp: PnP ACPI: found 6 devices [ 0.917800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.921716] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.924311] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.926161] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.928421] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.931274] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.934753] NET: Registered protocol family 2 [ 0.937411] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.942728] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.946797] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.953278] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.957195] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960979] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964800] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968202] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971659] NET: Registered protocol family 1 [ 0.975397] RPC: Registered named UNIX socket transport module. [ 0.978404] RPC: Registered udp transport module. [ 0.980817] RPC: Registered tcp transport module. [ 0.982795] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985433] NET: Registered protocol family 44 [ 0.987138] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989491] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992329] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995623] PCI: CLS 0 bytes, default 64 [ 0.997701] Unpacking initramfs... [ 2.410726] debug: unmapping init [mem 0xffff987abcc54000-0xffff987abffbffff] [ 2.415168] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.417825] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.421086] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.929510] Initialise system trusted keyrings [ 2.931217] Key type blacklist registered [ 2.933610] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.946307] zbud: loaded [ 2.950821] *** VALIDATE nfs *** [ 2.952284] *** VALIDATE nfs4 *** [ 2.954264] pstore: using deflate compression [ 2.970609] Platform Keyring initialized [ 3.077651] NET: Registered protocol family 38 [ 3.080201] Key type asymmetric registered [ 3.082392] Asymmetric key parser 'x509' registered [ 3.085380] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.089274] io scheduler mq-deadline registered [ 3.090860] io scheduler kyber registered [ 3.092519] io scheduler bfq registered [ 3.094633] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.097644] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.101243] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.104787] ACPI: Power Button [PWRF] [ 3.110823] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.118269] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.132960] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.142329] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.167536] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.195625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.225548] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.231133] Non-volatile memory driver v1.3 [ 3.232426] Linux agpgart interface v0.103 [ 3.272351] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.275630] vda: detected capacity change from 0 to 74756096 [ 3.292779] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.296976] vdb: detected capacity change from 0 to 1073741824 [ 3.312830] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.316320] vdc: detected capacity change from 0 to 2621440000 [ 3.333062] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.336425] vdd: detected capacity change from 0 to 2621440000 [ 3.354986] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.358290] vde: detected capacity change from 0 to 4294967296 [ 3.381946] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385384] vdf: detected capacity change from 0 to 4294967296 [ 3.394828] libphy: Fixed MDIO Bus: probed [ 3.405454] usbcore: registered new interface driver usbserial_generic [ 3.408919] usbserial: USB Serial support registered for generic [ 3.411870] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.416797] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.418976] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.421483] mousedev: PS/2 mouse device common for all mice [ 3.424919] rtc_cmos 00:05: RTC can wake from S4 [ 3.427736] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.428247] rtc_cmos 00:05: registered as rtc0 [ 3.433783] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.437391] intel_pstate: CPU model not supported [ 3.438503] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.443263] hid: raw HID events driver (C) Jiri Kosina [ 3.445494] usbcore: registered new interface driver usbhid [ 3.445864] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.447278] usbhid: USB HID core driver [ 3.447408] drop_monitor: Initializing network drop monitor service [ 3.454344] Initializing XFRM netlink socket [ 3.456499] NET: Registered protocol family 10 [ 3.459791] Segment Routing with IPv6 [ 3.461379] NET: Registered protocol family 17 [ 3.463585] mpls_gso: MPLS GSO support [ 3.468327] RAS: Correctable Errors collector initialized. [ 3.470661] AVX version of gcm_enc/dec engaged. [ 3.472365] AES CTR mode by8 optimization enabled [ 3.555331] sched_clock: Marking stable (3555311442, 0)->(4460590356, -905278914) [ 3.558756] registered taskstats version 1 [ 3.560526] Loading compiled-in X.509 certificates [ 3.564294] zswap: loaded using pool lzo/zbud [ 3.589167] Key type big_key registered [ 3.600435] Key type encrypted registered [ 3.602247] ima: No TPM chip found, activating TPM-bypass! [ 3.604643] ima: Allocated hash algorithm: sha1 [ 3.606468] ima: No architecture policies found [ 3.608449] evm: Initialising EVM extended attributes: [ 3.610718] evm: security.selinux [ 3.612079] evm: security.ima [ 3.613268] evm: security.capability [ 3.614789] evm: HMAC attrs: 0x1 [ 3.617272] rtc_cmos 00:05: setting system clock to 2026-08-06 19:25:57 UTC (1786044357) [ 3.624090] debug: unmapping init [mem 0xffffffffbe603000-0xffffffffbe7fffff] [ 3.627722] debug: unmapping init [mem 0xffffffffbd382000-0xffffffffbd658fff] [ 3.636225] Write protecting the kernel read-only data: 28672k [ 3.640229] debug: unmapping init [mem 0xffffffffbba03000-0xffffffffbbbfffff] [ 3.643252] debug: unmapping init [mem 0xffffffffbc314000-0xffffffffbc3fffff] [ 3.677706] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.686209] systemd[1]: Detected virtualization kvm. [ 3.688331] systemd[1]: Detected architecture x86-64. [ 3.690397] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.714823] systemd[1]: No hostname configured. [ 3.716821] systemd[1]: Set hostname to . [ 3.718956] random: systemd: uninitialized urandom read (16 bytes read) [ 3.720987] systemd[1]: Initializing machine ID from random generator. [ 3.849172] random: systemd: uninitialized urandom read (16 bytes read) [ 3.851706] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.855770] random: systemd: uninitialized urandom read (16 bytes read) [ 3.858537] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.863167] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.485566] device-mapper: uevent: version 1.0.3 [ 4.488274] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.139868] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.175000] virtio_net virtio0 ens2: renamed from eth0 [ 5.234219] scsi host0: ata_piix [ 5.263638] scsi host1: ata_piix [ 5.265281] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.269881] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.929961] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.122595] random: crng init done [ 10.124229] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.525243] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.612670] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.898871] SELinux: Disabled at runtime. [ 11.956635] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.965657] systemd[1]: Detected virtualization kvm. [ 11.967950] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.473971] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.476904] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.481306] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.485279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.487963] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.497664] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.503822] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ 12.554833] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.943117] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.215809] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.248196] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.423593] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.435769] EDAC sbridge: Ver: 1.1.2 [ 15.083312] Key type dns_resolver registered [ 15.388997] NFS: Registering the id_resolver key type [ 15.390985] Key type id_resolver registered [ 15.392700] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg602-server login: [ 82.783230] libcfs: loading out-of-tree module taints kernel. [ 82.830338] Key type ._llcrypt registered [ 82.831950] Key type .llcrypt registered [ 82.936848] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 99.703424] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 101.356083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.366447] alg: No test for adler32 (adler32-zlib) [ 102.941211] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 103.932938] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 105.785530] Key type lgssc registered [ 107.466130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.121213] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.680443] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 178.762363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.804836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.107353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.145773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.269523] Lustre: lustre-MDT0000: new disk, initializing [ 180.369824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.388519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.588874] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.841095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.966475] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.023620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.031458] Lustre: Skipped 1 previous similar message [ 201.149954] Lustre: lustre-MDT0001: new disk, initializing [ 201.238997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.287544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.299292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.343093] hrtimer: interrupt took 2434741 ns [ 206.131230] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.362253] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.141357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.387635] Lustre: lustre-OST0000: new disk, initializing [ 220.394961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.395852] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 220.401760] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.478222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.849314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.872190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.911032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 228.699980] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.639858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.780202] Lustre: lustre-OST0001: new disk, initializing [ 245.789463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.795375] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.882913] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.389394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.396754] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.453732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.580669] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.736293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.588444] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.358946] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.790600] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing yml_node [ 290.256574] Lustre: DEBUG MARKER: Client: 2.17.56.53 [ 292.560915] Lustre: DEBUG MARKER: MDS: 2.17.56.53 [ 295.213575] Lustre: DEBUG MARKER: OSS: 2.17.56.53 [ 296.721707] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 15:30:48 EDT 2026 [ 317.477667] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 318.983379] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.408597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.410920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.437643] Lustre: Skipped 2 previous similar messages [ 338.458138] Lustre: Skipped 3 previous similar messages [ 341.717238] Lustre: server umount lustre-MDT0000 complete [ 346.067713] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044699 with bad export cookie 2568327181833701245 [ 346.073030] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.079358] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 348.642091] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.643611] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.649207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.660207] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 348.708670] Lustre: Skipped 2 previous similar messages [ 351.653132] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.653613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.680200] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 351.706509] Lustre: Skipped 2 previous similar messages [ 352.516372] Lustre: server umount lustre-MDT0001 complete [ 356.839658] Lustre: server umount lustre-OST0000 complete [ 360.761267] Lustre: server umount lustre-OST0001 complete [ 367.636626] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 376.981209] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 415.197837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.461610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.500560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.593810] Lustre: lustre-MDT0000: new disk, initializing [ 415.709312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.726817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.750038] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 432.378363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.491352] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.575404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.581318] Lustre: Skipped 1 previous similar message [ 432.701724] Lustre: lustre-MDT0001: new disk, initializing [ 432.867425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.930208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.953491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.180689] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 447.663616] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.721241] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 461.762345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.276207] Lustre: lustre-OST0000: new disk, initializing [ 462.280813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.292377] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.433152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.384963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.407608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.609240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.297430] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 484.892659] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.407458] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 490.653055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 495.259366] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 495.484354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.709609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.711705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.729661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.731228] LustreError: Skipped 1 previous similar message [ 500.736570] Lustre: Skipped 1 previous similar message [ 503.615966] Lustre: server umount lustre-OST0000 complete [ 509.921661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.929175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.939469] Lustre: Skipped 1 previous similar message [ 509.947195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.876363] Lustre: server umount lustre-MDT0000 complete [ 516.072235] LustreError: 17998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.083212] LustreError: 17998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 517.576397] LustreError: 15880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044871 with bad export cookie 2568327181833702778 [ 517.587580] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.598080] LustreError: 15880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 517.903551] Lustre: server umount lustre-MDT0001 complete [ 523.919464] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:34:35 (1786044875) [ 549.054608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.465462] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.517192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.829545] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 554.981330] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.516188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.704286] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 576.093888] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.236308] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.836752] Lustre: Failing over lustre-MDT0000 [ 586.357844] Lustre: server umount lustre-MDT0000 complete [ 587.947960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.962245] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.980815] Lustre: Skipped 2 previous similar messages [ 588.031831] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 604.644560] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786044941/real 1786044941] req@ffff987b1bd9e680 x1872803352708480/t0(0) o400->MGC192.168.206.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786044957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 604.670938] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.690617] LustreError: 21277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.716597] LustreError: 21277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 608.674493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.313072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.325755] Lustre: Skipped 1 previous similar message [ 616.432643] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 621.541526] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.551229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.572353] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 621.734918] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 631.950475] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 636.900643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.924857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.949231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 636.960917] Lustre: Skipped 2 previous similar messages [ 642.026792] Lustre: server umount lustre-MDT0000 complete [ 643.049795] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.088644] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 646.155153] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045000 with bad export cookie 2568327181833704192 [ 646.164694] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.179450] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 646.484960] Lustre: server umount lustre-MDT0001 complete [ 655.646087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.046685] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 671.523376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.491984] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 684.001520] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 689.848913] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 692.193846] Lustre: Failing over lustre-MDT0000 [ 692.208159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.211293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.227412] Lustre: Skipped 2 previous similar messages [ 692.231366] Lustre: Skipped 2 previous similar messages [ 692.452920] Lustre: server umount lustre-MDT0000 complete [ 711.655170] LustreError: 23257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.671340] LustreError: 23257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 711.987113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.106828] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.434651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.469366] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 716.498760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.506641] Lustre: Skipped 1 previous similar message [ 716.532200] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 716.621503] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 727.210622] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 732.129316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 732.136497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.602871] Lustre: server umount lustre-MDT0000 complete [ 741.069037] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045094 with bad export cookie 2568327181833705130 [ 741.081260] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.409617] Lustre: server umount lustre-MDT0001 complete [ 750.877974] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:38:22 (1786045102) [ 768.652150] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 779.726312] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 822.661163] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 833.805335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.131826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 834.161063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 834.243230] Lustre: lustre-MDT0000: new disk, initializing [ 834.365114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.379535] Lustre: Skipped 3 previous similar messages [ 834.397458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 839.197582] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 851.055440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.224326] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 851.549381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 856.268684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 866.434407] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 872.589588] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.894836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.251326] Lustre: 31608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 882.152211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 882.202045] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 882.269969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 882.281636] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 887.577299] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 897.337760] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 904.661561] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 928.241018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.242755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.252083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 928.266780] Lustre: Skipped 2 previous similar messages [ 928.272807] Lustre: Skipped 4 previous similar messages [ 934.615550] Lustre: server umount lustre-OST0000 complete [ 943.590789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 943.596389] LustreError: Skipped 1 previous similar message [ 945.170856] Lustre: server umount lustre-MDT0000 complete [ 948.717470] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.738883] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 949.568908] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045303 with bad export cookie 2568327181833705536 [ 949.580538] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.596045] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 950.328433] Lustre: server umount lustre-MDT0001 complete [ 962.423231] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 965.609656] Key type lgssc unregistered [ 965.929721] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.937460] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.961393] LNet: Removed LNI 192.168.206.102@tcp [ 966.991209] Key type .llcrypt unregistered [ 966.997294] Key type ._llcrypt unregistered [ 993.607505] Key type ._llcrypt registered [ 993.609940] Key type .llcrypt registered [ 993.744451] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 995.124078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.155802] alg: No test for adler32 (adler32-zlib) [ 996.259349] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 996.526329] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 998.176498] Key type lgssc registered [ 999.354806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.891558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.919301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.461199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.102982] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1019.878873] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.903367] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1022.450961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.746627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.750982] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1033.686962] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.882792] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1045.508640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.825810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.928824] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1051.346497] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1058.582282] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.745856] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:43:43 (1786045423) [ 1073.556617] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1075.282919] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1081.399505] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:43:53 (1786045433) [ 1089.277830] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:44:01 (1786045441) [ 1090.080780] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1097.935981] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1097.943485] Lustre: Skipped 2 previous similar messages [ 1099.515210] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:44:11 (1786045451) [ 1103.236667] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1103.240457] Lustre: Skipped 3 previous similar messages [ 1119.425766] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1119.434940] Lustre: Skipped 6 previous similar messages [ 1122.149824] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:44:34 (1786045474) [ 1124.766363] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1128.354774] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1136.880658] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:44:48 (1786045488) [ 1138.768359] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1138.775896] Lustre: Skipped 3 previous similar messages [ 1150.596525] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:45:02 (1786045502) [ 1165.793411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.800494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.815717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.842544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.845530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.872961] Lustre: Skipped 1 previous similar message [ 1170.205266] Lustre: server umount lustre-OST0000 complete [ 1176.546055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1176.552990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.567800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.201405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.210393] Lustre: Skipped 3 previous similar messages [ 1188.321810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.324852] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1188.335812] Lustre: Skipped 1 previous similar message [ 1188.550717] Lustre: server umount lustre-MDT0000 complete [ 1191.673535] LustreError: 38568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045545 with bad export cookie 3176019659834592401 [ 1191.689551] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.695503] LustreError: 38568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1192.039552] Lustre: server umount lustre-MDT0001 complete [ 1202.631596] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1205.627586] Key type lgssc unregistered [ 1205.999941] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.014690] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.044297] LNet: Removed LNI 192.168.206.102@tcp [ 1206.854738] Key type .llcrypt unregistered [ 1206.856723] Key type ._llcrypt unregistered [ 1225.686329] Key type ._llcrypt registered [ 1225.689062] Key type .llcrypt registered [ 1225.758436] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 1240.058948] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1240.798766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1240.825905] alg: No test for adler32 (adler32-zlib) [ 1241.829793] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 1242.055648] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1243.728430] Key type lgssc registered [ 1244.780137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1279.194919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1279.224937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.489365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1280.525694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1280.647155] Lustre: lustre-MDT0000: new disk, initializing [ 1280.780430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.809237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1285.260654] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1295.986959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.148961] Lustre: 46010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1296.222780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1296.227527] Lustre: Skipped 1 previous similar message [ 1296.384128] Lustre: lustre-MDT0001: new disk, initializing [ 1296.534596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.595573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1296.606912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1301.528749] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1310.284485] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.249550] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.413417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1321.659775] Lustre: lustre-OST0000: new disk, initializing [ 1321.663552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1321.669195] Lustre: 47981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1321.733649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.035553] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1329.241381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1329.255373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1329.390713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1338.059711] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1342.837956] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1343.025107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1347.798391] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1348.039226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1350.131843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.142741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1350.151842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.173114] Lustre: Skipped 1 previous similar message [ 1355.238584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.247164] Lustre: Skipped 1 previous similar message [ 1356.309845] Lustre: server umount lustre-OST0000 complete [ 1360.357341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.365027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.379643] Lustre: Skipped 2 previous similar messages [ 1360.391072] Lustre: Skipped 1 previous similar message [ 1365.474439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.481641] Lustre: Skipped 1 previous similar message [ 1366.477648] Lustre: server umount lustre-MDT0000 complete [ 1369.828991] LustreError: 47980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045723 with bad export cookie 12431684542232087673 [ 1369.833061] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1369.845477] LustreError: 47980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1370.192601] Lustre: server umount lustre-MDT0001 complete [ 1379.505129] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:48:51 (1786045731) [ 1386.698035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.445487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.055666] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1392.614486] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.638971] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1397.732235] LustreError: 50441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.857572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.289889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1407.519904] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1415.513695] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1422.044437] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1428.820174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.204450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.470569] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1444.797874] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.707114] LustreError: 52969:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1453.466244] Lustre: Modifying parameter general.timeout=1 in log params [ 1454.940889] Lustre: Modifying parameter general.timeout=3 in log params [ 1454.946271] Lustre: Skipped 1 previous similar message [ 1457.712887] Lustre: Modifying parameter general.timeout=6 in log params [ 1457.715453] Lustre: Skipped 2 previous similar messages [ 1462.323930] Lustre: Modifying parameter general.timeout=11 in log params [ 1462.332703] Lustre: Skipped 4 previous similar messages [ 1471.096700] Lustre: Modifying parameter general.timeout=21 in log params [ 1471.105864] Lustre: Skipped 9 previous similar messages [ 1487.548714] Lustre: Modifying parameter general.timeout=39 in log params [ 1487.563205] Lustre: Skipped 17 previous similar messages [ 1520.083567] Lustre: Modifying parameter general.timeout=73 in log params [ 1520.087119] Lustre: Skipped 33 previous similar messages [ 1583.437883] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:52:15 (1786045935) [ 1584.492065] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1584.496641] Lustre: Skipped 57 previous similar messages [ 1586.179863] Lustre: 59914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1588.009966] Lustre: 60012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1591.152450] Lustre: 60223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1591.162641] Lustre: 60223:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1626.346321] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1626.360100] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1626.387690] LustreError: 61801:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1630.370952] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:53:01 (1786045981) [ 1646.052562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.058088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.065340] Lustre: Skipped 2 previous similar messages [ 1646.070749] Lustre: Skipped 1 previous similar message [ 1649.823727] Lustre: server umount lustre-MDT0000 complete [ 1651.174022] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.209672] LustreError: 50440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1653.752025] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046007 with bad export cookie 12431684542232088562 [ 1653.759303] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.771946] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1654.076367] Lustre: server umount lustre-MDT0001 complete [ 1668.454282] Lustre: server umount lustre-OST0000 complete [ 1697.646896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.078930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.332840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.697180] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1712.480590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.577877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1712.598543] Lustre: 64334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1718.254596] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1726.612867] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1733.585484] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1740.836966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1741.038465] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1741.436743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.446540] Lustre: Skipped 1 previous similar message [ 1748.330420] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1755.982162] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.667406] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1757.697391] Lustre: Skipped 22 previous similar messages [ 1757.773513] Lustre: 66899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1768.431559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1768.436486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.443633] LustreError: Skipped 1 previous similar message [ 1768.449471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1768.451049] Lustre: Skipped 1 previous similar message [ 1768.460331] Lustre: Skipped 1 previous similar message [ 1772.047205] Lustre: server umount lustre-OST0000 complete [ 1778.657347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.115794] Lustre: server umount lustre-MDT0000 complete [ 1783.779205] LustreError: 65598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.797234] LustreError: 65598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1785.911863] LustreError: 64327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046139 with bad export cookie 12431684542232091705 [ 1785.913256] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.923528] LustreError: 64327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1786.129685] Lustre: server umount lustre-MDT0001 complete [ 1800.473160] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1803.225160] Key type lgssc unregistered [ 1803.647173] LNet: 68115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.657673] LNetError: 68115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.716797] LNet: Removed LNI 192.168.206.102@tcp [ 1806.126368] Key type .llcrypt unregistered [ 1806.129310] Key type ._llcrypt unregistered [ 1830.874690] Key type ._llcrypt registered [ 1830.877107] Key type .llcrypt registered [ 1830.991663] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1831.688181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.736207] alg: No test for adler32 (adler32-zlib) [ 1832.783613] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 1833.021122] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1834.672170] Key type lgssc registered [ 1835.637717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1842.560731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1842.570861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.063383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.795437] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1853.417532] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.435690] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1857.898114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.495573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.393717] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1870.945162] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.468853] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1884.626180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1885.144500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.817896] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1899.901590] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1911.700156] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1925.090140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.094318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.097970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.101303] Lustre: Skipped 1 previous similar message [ 1926.731226] Lustre: server umount lustre-MDT0000 complete [ 1930.211163] LustreError: 69527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.250249] LustreError: 69527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1930.311676] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046284 with bad export cookie 906163191814625419 [ 1930.329486] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1930.332389] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.599104] Lustre: server umount lustre-MDT0001 complete [ 1944.978333] Lustre: server umount lustre-OST0000 complete [ 1973.572154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.787760] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1973.813954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.079176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.334578] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1991.123837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.573717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1991.650959] Lustre: 74401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1992.551966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.672978] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2006.467370] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.167940] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.510867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2019.637519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2019.988483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.863772] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2036.203670] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.889532] Lustre: 76975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.993551] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2046.951810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2046.969785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.987940] Lustre: Skipped 1 previous similar message [ 2046.999424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.008871] Lustre: Skipped 2 previous similar messages [ 2051.997583] Lustre: server umount lustre-OST0000 complete [ 2057.707375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.733555] Lustre: Skipped 1 previous similar message [ 2057.749415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.761908] Lustre: Skipped 3 previous similar messages [ 2061.852840] Lustre: server umount lustre-MDT0000 complete [ 2062.818753] LustreError: 74411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.843695] LustreError: 74411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2065.977412] LustreError: 76267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046419 with bad export cookie 906163191814626266 [ 2065.995071] LustreError: 76267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2065.999933] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.018291] LustreError: 68841:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 2066.532762] Lustre: server umount lustre-MDT0001 complete [ 2077.596525] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2080.566658] Key type lgssc unregistered [ 2080.872569] LNet: 78190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.888370] LNetError: 78190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.909476] LNet: Removed LNI 192.168.206.102@tcp [ 2082.008304] Key type .llcrypt unregistered [ 2082.009683] Key type ._llcrypt unregistered [ 2107.479850] Key type ._llcrypt registered [ 2107.481629] Key type .llcrypt registered [ 2107.635092] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2108.935328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.953135] alg: No test for adler32 (adler32-zlib) [ 2109.907679] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 2110.070828] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2111.689972] Key type lgssc registered [ 2112.957535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2121.240784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2121.289638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.870264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.825816] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2132.470227] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.495432] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2136.818125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.468084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.357603] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2151.192760] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2156.812441] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2163.186961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.731931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.016534] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2177.237279] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2186.300400] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2187.276736] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.938973] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.948427] Lustre: Skipped 1 previous similar message [ 2191.458157] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2191.461983] Lustre: Skipped 2 previous similar messages [ 2195.798980] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.811166] Lustre: Skipped 4 previous similar messages [ 2204.548165] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2204.556731] Lustre: Skipped 9 previous similar messages [ 2221.100297] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2221.118319] Lustre: Skipped 17 previous similar messages [ 2253.912248] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2253.919580] Lustre: Skipped 36 previous similar messages [ 2287.381511] Lustre: 87719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2290.369228] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:04:02 (1786046642) [ 2291.985284] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2294.106761] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:04:05 (1786046645) [ 2299.874619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.892819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.901341] Lustre: Skipped 1 previous similar message [ 2304.998391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.008359] Lustre: Skipped 2 previous similar messages [ 2305.235932] Lustre: server umount lustre-OST0000 complete [ 2310.113905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.119207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.136029] Lustre: Skipped 1 previous similar message [ 2310.174290] Lustre: Skipped 1 previous similar message [ 2315.234673] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.255743] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2315.358499] Lustre: server umount lustre-MDT0000 complete [ 2318.989508] LustreError: 79597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046672 with bad export cookie 84867642029299982 [ 2318.993975] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.004027] LustreError: 79597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2319.375516] Lustre: server umount lustre-MDT0001 complete [ 2331.065531] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2333.411157] Key type lgssc unregistered [ 2333.711820] LNet: 88971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.725147] LNetError: 88971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.754700] LNet: Removed LNI 192.168.206.102@tcp [ 2334.691037] Key type .llcrypt unregistered [ 2334.693560] Key type ._llcrypt unregistered [ 2339.965557] Key type ._llcrypt registered [ 2339.967706] Key type .llcrypt registered [ 2340.076821] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2345.061179] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules [ 2345.868885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.908569] alg: No test for adler32 (adler32-zlib) [ 2346.999203] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 2347.260122] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2348.936185] Key type lgssc registered [ 2349.955260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.921378] LustreError: 90190:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2367.660150] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2372.964240] LustreError: 90190:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2376.663828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.692555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.284542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.890528] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2388.456073] LustreError: 91214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.474970] LustreError: 91214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2391.545886] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:05:43 (1786046743) [ 2393.574085] LustreError: 91213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.878333] LustreError: 91245:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2395.886715] LustreError: 91245:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2395.901141] LustreError: 91245:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2396.298787] Lustre: server umount lustre-MDT0000 complete [ 2411.135841] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2413.992553] Key type lgssc unregistered [ 2414.295199] LNet: 92739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.320173] LNetError: 92739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.348458] LNet: Removed LNI 192.168.206.102@tcp [ 2415.225262] Key type .llcrypt unregistered [ 2415.228064] Key type ._llcrypt unregistered [ 2432.574486] Key type ._llcrypt registered [ 2432.576943] Key type .llcrypt registered [ 2432.700136] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2433.856875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.900505] alg: No test for adler32 (adler32-zlib) [ 2434.997780] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 2435.217244] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2436.872229] Key type lgssc registered [ 2437.739529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2444.022281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2444.042097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.494875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.091546] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2456.043664] LustreError: 93958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.058388] LustreError: 93958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2457.572326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.336112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.592773] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2470.171902] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2475.917847] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.372490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.489088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.062679] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2497.069861] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.066894] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:08:16 (1786046896) [ 2546.676460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2548.545335] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:08:20 (1786046900) [ 2554.343260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.348314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.366331] Lustre: Skipped 2 previous similar messages [ 2554.385346] Lustre: Skipped 2 previous similar messages [ 2559.253281] Lustre: server umount lustre-MDT0000 complete [ 2559.469943] LustreError: 93952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.493240] LustreError: 93952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2563.401258] LustreError: 97206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046917 with bad export cookie 3118451755073963993 [ 2563.403155] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2563.410791] LustreError: 97206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2563.658264] Lustre: server umount lustre-MDT0001 complete [ 2577.988966] Lustre: server umount lustre-OST0000 complete [ 2590.050980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.619230] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.852207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.351317] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2596.332430] LustreError: 98238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.429272] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.150248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.699847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.807572] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2616.275891] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.075702] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2635.358314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2635.503688] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2635.508184] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2635.530273] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2635.536084] LustreError: 100320:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2635.541373] LustreError: 100320:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2635.546353] LustreError: 100320:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2635.613837] Lustre: server umount lustre-OST0000 complete [ 2635.616078] LustreError: 100320:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2643.817778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.026537] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2644.039166] LustreError: 98227:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2644.074031] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2644.079625] LustreError: 100787:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2644.091380] LustreError: 100787:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2644.104994] LustreError: 100787:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2644.187376] Lustre: server umount lustre-OST0000 complete [ 2644.194531] LustreError: 100787:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2654.295142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.571545] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2654.575765] Lustre: Found index 0 for lustre-OST0000, updating log [ 2654.581259] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2654.669195] Lustre: lustre-OST0000: new disk, initializing [ 2654.673984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.688536] Lustre: 101321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2655.161995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.649435] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2665.821933] Lustre: Failing over lustre-OST0000 [ 2666.032353] Lustre: server umount lustre-OST0000 complete [ 2670.561528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.594632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.606279] Lustre: Skipped 1 previous similar message [ 2674.727463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.737529] Lustre: Skipped 1 previous similar message [ 2679.780688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.799088] Lustre: Skipped 1 previous similar message [ 2684.384139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2684.716384] Lustre: server umount lustre-MDT0000 complete [ 2684.903711] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.937235] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2688.788771] LustreError: 98217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047042 with bad export cookie 3118451755073966737 [ 2688.798930] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.804876] LustreError: 98217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2689.294103] Lustre: server umount lustre-MDT0001 complete [ 2698.338681] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:10:50 (1786047050) [ 2714.887582] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2724.771427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.144940] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.254332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.291882] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2737.900414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.784753] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2745.792290] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2753.439405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.493105] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2761.598670] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2763.683252] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2763.698608] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2763.714153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2768.897879] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2768.923102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.004880] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2771.050627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.079323] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2771.079512] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2771.096913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2773.027463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.294070] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2773.482768] Lustre: lustre-OST0001: new disk, initializing [ 2773.493335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2773.522434] Lustre: 105990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2773.888279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2773.891600] Lustre: Skipped 2 previous similar messages [ 2780.799321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2780.804876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2780.969968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2782.809337] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2792.106388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2803.243632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2810.087836] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.2@tcp (stopping) [ 2810.341929] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.352863] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.363272] Lustre: Skipped 1 previous similar message [ 2813.962664] Lustre: server umount lustre-MDT0001 complete [ 2815.176466] LustreError: 103664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.190724] LustreError: 103664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2823.005756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.128570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2823.134588] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2823.202729] Lustre: 103657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2824.879062] LustreError: 107537:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2824.882340] LustreError: 107537:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2824.889726] LustreError: 103660:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2824.986131] LustreError: 107558:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.102@tcp: cfg command failed: rc = -17 [ 2824.994302] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2824.994302] [ 2828.420580] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2829.314060] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2838.915248] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:13:10 (1786047190) [ 2856.233178] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2874.009497] Lustre: 109603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2897.197612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2900.694366] Lustre: 110738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2991.073583] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.074424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.096095] Lustre: Skipped 4 previous similar messages [ 2991.100888] Lustre: Skipped 5 previous similar messages [ 2994.204627] Lustre: server umount lustre-MDT0000 complete [ 2996.197316] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.213882] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2997.502308] LustreError: 105186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047351 with bad export cookie 3118451755073967738 [ 2997.508294] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.517468] LustreError: 105186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2997.802944] Lustre: server umount lustre-MDT0001 complete [ 3011.978236] Lustre: server umount lustre-OST0000 complete [ 3026.077121] Lustre: server umount lustre-OST0001 complete [ 3031.803577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3041.582520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3050.079313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3059.121943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3077.723481] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3087.886130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.906265] Lustre: lustre-MDT0000: reset Object Index mappings [ 3088.525929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.535967] Lustre: Skipped 1 previous similar message [ 3093.401651] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3102.125251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.138117] Lustre: lustre-MDT0001: reset Object Index mappings [ 3106.905195] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3109.595769] Lustre: 115844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3115.878390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.877363] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3127.735404] LustreError: 116196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.759960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3127.761211] LustreError: 116196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3130.361917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.007414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3134.008391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3134.031300] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3136.755929] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3144.640286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3148.450948] Lustre: 117718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3199.856368] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:19:11 (1786047551) [ 3229.158760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.162711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.171740] Lustre: Skipped 3 previous similar messages [ 3229.197825] Lustre: Skipped 5 previous similar messages [ 3232.432122] Lustre: server umount lustre-MDT0000 complete [ 3236.190557] LustreError: 114677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047590 with bad export cookie 3118451755074014813 [ 3236.198117] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.200472] LustreError: 114677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.397153] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.408462] Lustre: Skipped 2 previous similar messages [ 3242.640727] Lustre: server umount lustre-MDT0001 complete [ 3252.531897] Lustre: server umount lustre-OST0000 complete [ 3270.113861] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3270.522394] Lustre: server umount lustre-OST0001 complete [ 3277.908557] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 3289.092212] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3327.781128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.133255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3328.185604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3328.292828] Lustre: lustre-MDT0000: new disk, initializing [ 3328.412814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.423564] Lustre: Skipped 3 previous similar messages [ 3328.444469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3333.491996] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3340.687449] Lustre: server umount lustre-MDT0000 complete [ 3350.967785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.286865] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3351.310188] Lustre: 121985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3351.336143] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3356.119499] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3361.313234] Lustre: server umount lustre-MDT0000 complete [ 3370.299913] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:22:01 (1786047721) [ 3372.273584] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3374.069423] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:22:06 (1786047726) [ 3391.757320] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3401.333818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.329330] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3413.999515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.081858] Lustre: 124037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3414.121768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3414.125716] Lustre: Skipped 1 previous similar message [ 3414.177341] Lustre: lustre-MDT0001: new disk, initializing [ 3414.281232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3414.293087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3418.252624] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3422.719631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3429.146742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.368778] Lustre: lustre-OST0000: new disk, initializing [ 3429.371546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3429.376546] Lustre: 125615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3430.623978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3430.635091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3430.753987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3435.623498] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3445.675823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.880929] Lustre: lustre-OST0001: new disk, initializing [ 3445.887163] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3445.893931] Lustre: 126486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3447.348088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3447.356868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3447.465826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3451.877525] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3461.544598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3465.039270] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3846.539374] Lustre: 132909:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3846.546490] Lustre: 132909:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3846.555875] Lustre: 132909:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3846.559324] Lustre: 132909:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3846.563260] Lustre: 132909:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3846.566880] Lustre: 132909:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4171.968461] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:35:23 (1786048523) [ 4187.629483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.635968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.652494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.659366] Lustre: Skipped 4 previous similar messages [ 4190.181997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.202956] Lustre: Skipped 1 previous similar message [ 4193.790484] Lustre: server umount lustre-MDT0000 complete [ 4195.304729] LustreError: 124043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.340280] LustreError: 124043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4198.312451] LustreError: 124027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786048552 with bad export cookie 3118451755074082888 [ 4198.320338] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.328740] LustreError: 124027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4200.432964] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.436415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4200.446200] Lustre: Skipped 2 previous similar messages [ 4200.455095] Lustre: Skipped 4 previous similar messages [ 4204.942889] Lustre: server umount lustre-MDT0001 complete [ 4215.236228] Lustre: server umount lustre-OST0000 complete [ 4225.310060] Lustre: server umount lustre-OST0001 complete [ 4231.029364] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 4238.430164] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 4267.522133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.760699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4267.789495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4267.883630] Lustre: lustre-MDT0000: new disk, initializing [ 4267.957640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.964126] Lustre: Skipped 5 previous similar messages [ 4267.996082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4272.053601] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4283.605577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.774175] Lustre: 135970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4283.849459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4283.853656] Lustre: Skipped 1 previous similar message [ 4283.907486] Lustre: lustre-MDT0001: new disk, initializing [ 4283.996959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.008495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4289.404564] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4299.990514] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4305.576653] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4312.353287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.642474] Lustre: lustre-OST0000: new disk, initializing [ 4312.645924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4312.651254] Lustre: 137942:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4312.741684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.748071] Lustre: Skipped 1 previous similar message [ 4314.403258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4314.421018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4314.558072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4319.055309] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4329.561719] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4340.704684] Lustre: lustre-MDD0000: changelog on [ 4343.884102] Lustre: lustre-MDD0001: changelog on [ 6481.511609] Lustre: lustre-MDD0001: changelog off [ 6483.434933] Lustre: 139754:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6484.506828] Lustre: lustre-MDD0000: changelog off [ 6490.596045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.603244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.609608] LustreError: Skipped 1 previous similar message [ 6490.628868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.638064] Lustre: Skipped 1 previous similar message [ 6490.657923] Lustre: Skipped 4 previous similar messages [ 6495.720805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6495.730243] Lustre: Skipped 1 previous similar message [ 6496.756311] Lustre: server umount lustre-OST0000 complete [ 6500.835823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6500.847912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.852691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.870301] Lustre: Skipped 1 previous similar message [ 6505.953995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6505.963466] Lustre: Skipped 2 previous similar messages [ 6515.168961] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6515.531254] Lustre: server umount lustre-MDT0000 complete [ 6516.198230] LustreError: 136856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.218257] LustreError: 136856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6519.816891] LustreError: 135961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786050873 with bad export cookie 3118451755074900887 [ 6519.826690] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.834812] LustreError: 135961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6520.272355] Lustre: server umount lustre-MDT0001 complete [ 6531.393134] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 6534.105926] Key type lgssc unregistered [ 6534.527552] LNet: 147146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6534.543924] LNetError: 147146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6534.575299] LNet: Removed LNI 192.168.206.102@tcp [ 6535.482238] Key type .llcrypt unregistered [ 6535.486149] Key type ._llcrypt unregistered [ 6557.575568] Key type ._llcrypt registered [ 6557.578909] Key type .llcrypt registered [ 6557.688946] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 6574.657444] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 6575.773917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6575.873069] alg: No test for adler32 (adler32-zlib) [ 6576.909639] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 6577.214683] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 6578.936823] Key type lgssc registered [ 6580.238862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6619.335555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6619.426119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.684516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.726724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6620.850696] Lustre: lustre-MDT0000: new disk, initializing [ 6620.958571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.980790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6625.985268] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6638.036556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.190695] Lustre: 150349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6638.283283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6638.290832] Lustre: Skipped 1 previous similar message [ 6638.447217] Lustre: lustre-MDT0001: new disk, initializing [ 6638.621293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.680242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6638.693683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6643.408866] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6653.846112] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6661.501410] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6670.473248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.768217] Lustre: lustre-OST0000: new disk, initializing [ 6670.779625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6670.795863] Lustre: 152318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6670.895387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.191792] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6680.643185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6680.657475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6680.804373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6688.721351] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.595343] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_module obdecho/obdecho [ 6699.164061] LustreError: 153322:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6699.184190] LustreError: 153322:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6701.517576] LustreError: 153419:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6715.361570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6715.371108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.384284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6716.909210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6716.912157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.922787] Lustre: Skipped 1 previous similar message [ 6716.959479] Lustre: Skipped 1 previous similar message [ 6721.486975] Lustre: server umount lustre-MDT0000 complete [ 6726.187850] LustreError: 150344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051080 with bad export cookie 9097901606844242556 [ 6726.191858] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.196120] LustreError: 150344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6726.560425] Lustre: server umount lustre-MDT0001 complete [ 6741.949796] Lustre: server umount lustre-OST0000 complete [ 6768.994717] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 6781.101758] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 6822.131616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.488718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6822.523727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6822.654488] Lustre: lustre-MDT0000: new disk, initializing [ 6822.751042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.776184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6828.167576] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6840.476256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.636225] Lustre: 157139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6840.703273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6840.709118] Lustre: Skipped 1 previous similar message [ 6840.799551] Lustre: lustre-MDT0001: new disk, initializing [ 6840.937979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6841.002115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6841.017970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6846.876909] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6856.806422] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6862.737122] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.383785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.769928] Lustre: lustre-OST0000: new disk, initializing [ 6869.776224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6869.783749] Lustre: 159106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6869.910805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.597260] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6877.751608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6877.766746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6877.874163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6884.873099] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6889.627684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6889.924648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6895.171799] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6895.512893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6898.661819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6898.665818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.668681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6898.677975] Lustre: Skipped 1 previous similar message [ 6898.680786] Lustre: Skipped 1 previous similar message [ 6903.780448] LustreError: 159572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.814963] LustreError: 159572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6903.959325] Lustre: server umount lustre-OST0000 complete [ 6907.363038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6907.374271] LustreError: Skipped 1 previous similar message [ 6907.380039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.404738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.411299] Lustre: Skipped 1 previous similar message [ 6913.698367] Lustre: server umount lustre-MDT0000 complete [ 6914.017610] LustreError: 158299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.026648] LustreError: 158299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6917.144824] LustreError: 157132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051271 with bad export cookie 9097901606844243515 [ 6917.147248] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6917.153079] LustreError: 157132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6917.535509] Lustre: server umount lustre-MDT0001 complete [ 6921.193829] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:21:13 (1786051273) [ 6923.025987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6925.037975] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:21:16 (1786051276) [ 6932.039872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.586421] LustreError: 161520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.661486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.068402] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6938.089559] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.248449] LustreError: 161520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.587562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.609721] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6958.926977] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6964.991903] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6971.828332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6972.134892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6972.142961] Lustre: Skipped 1 previous similar message [ 6977.636253] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6984.714332] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6996.180101] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 6996.187578] Lustre: Skipped 2 previous similar messages [ 6997.293968] Lustre: server umount lustre-MDT0000 complete [ 6997.473509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6997.489135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.510263] Lustre: Skipped 1 previous similar message [ 6997.536814] LustreError: 162248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.942970] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051354 with bad export cookie 9097901606844244397 [ 7000.952569] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7000.953563] LustreError: 161501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7005.095958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7005.109138] Lustre: Skipped 2 previous similar messages [ 7015.392519] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7015.396285] LustreError: 162255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.434125] LustreError: 162255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7015.711084] Lustre: server umount lustre-MDT0001 complete [ 7022.268888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7034.819855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.197197] LustreError: 165362:0:(llog.c:1655:llog_backup()) MGC192.168.206.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7035.206559] Lustre: 165362:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7051.233618] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.251663] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7095.969420] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7095.976094] Lustre: 165362:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7115.748867] LustreError: 165369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.765299] LustreError: 165369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7136.757657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7141.876663] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7142.783496] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7151.803664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.536771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7170.545927] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7e423d4f05683aea [ 7170.560852] Lustre: MGC192.168.206.102@tcp: Connection restored to 0@lo (at 0@lo) [ 7175.669486] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7176.180907] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7183.343760] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.014516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7190.717647] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7192.136910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7212.004809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7212.009457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.012122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7212.012128] Lustre: Skipped 4 previous similar messages [ 7212.052752] Lustre: Skipped 4 previous similar messages [ 7217.467388] Lustre: server umount lustre-MDT0000 complete [ 7221.828936] LustreError: 164300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051575 with bad export cookie 9097901606844250858 [ 7221.837625] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.852457] LustreError: 164300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7222.213953] Lustre: server umount lustre-MDT0001 complete [ 7237.000901] Lustre: server umount lustre-OST0000 complete [ 7259.920377] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 7270.846984] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7307.661750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.897194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7307.923834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7308.022625] Lustre: lustre-MDT0000: new disk, initializing [ 7308.119936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.127884] Lustre: Skipped 1 previous similar message [ 7308.147323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7312.205670] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7322.739109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.848267] Lustre: 170475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7323.113603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7327.506920] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7336.955524] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7342.147546] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7348.467963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.693364] Lustre: lustre-OST0000: new disk, initializing [ 7348.696085] Lustre: Skipped 1 previous similar message [ 7348.699249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7348.702856] Lustre: Skipped 2 previous similar messages [ 7348.707537] Lustre: 172443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7355.115190] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7355.435517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7355.445363] Lustre: Skipped 1 previous similar message [ 7355.461354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7355.596494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7363.119581] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.622222] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7366.792675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7371.081323] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7371.295431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7376.360326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.367599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7376.370333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.370342] Lustre: Skipped 5 previous similar messages [ 7376.373499] Lustre: Skipped 1 previous similar message [ 7376.398352] LustreError: Skipped 1 previous similar message [ 7379.172055] Lustre: server umount lustre-OST0000 complete [ 7388.672807] Lustre: server umount lustre-MDT0000 complete [ 7391.626066] LustreError: 170465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051745 with bad export cookie 9097901606844267700 [ 7391.632115] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7391.645724] LustreError: 170465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7391.896319] Lustre: server umount lustre-MDT0001 complete [ 7400.461220] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:29:12 (1786051752) [ 7407.057294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.541389] LustreError: 174853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.560368] LustreError: 174853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7411.384178] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7418.888847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.468725] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7429.783703] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7434.169892] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7439.214629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.231937] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7452.393557] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.802441] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:30:18 (1786051818) [ 7470.566074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7470.576558] LustreError: Skipped 1 previous similar message [ 7470.581148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.597057] Lustre: Skipped 2 previous similar messages [ 7470.605587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7470.609610] Lustre: Skipped 4 previous similar messages [ 7476.600687] Lustre: server umount lustre-OST0000 complete [ 7486.459729] Lustre: server umount lustre-MDT0000 complete [ 7490.285938] LustreError: 174833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051844 with bad export cookie 9097901606844268575 [ 7490.297305] LustreError: 174833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7490.297958] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.853302] Lustre: server umount lustre-MDT0001 complete [ 7502.549213] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7505.184925] Key type lgssc unregistered [ 7505.533740] LNet: 178585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7505.542408] LNetError: 178585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7505.570383] LNet: Removed LNI 192.168.206.102@tcp [ 7506.506923] Key type .llcrypt unregistered [ 7506.512274] Key type ._llcrypt unregistered [ 7524.159234] Key type ._llcrypt registered [ 7524.162479] Key type .llcrypt registered [ 7524.260197] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7525.393763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7525.489560] alg: No test for adler32 (adler32-zlib) [ 7526.478114] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 7526.801211] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 7528.504192] Key type lgssc registered [ 7529.572736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.794197] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7536.817347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7552.352393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7557.472963] LustreError: 179836:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.102@tcp: failed processing log, type 4: rc = -110 [ 7562.592638] LustreError: 179836:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7562.606977] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7562.617341] LustreError: MGC192.168.206.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7562.629372] LustreError: 179836:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7562.638300] LustreError: 179836:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7562.643653] LustreError: 179836:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7562.696318] Lustre: server umount lustre-OST0000 complete [ 7562.701215] LustreError: 179836:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7570.070860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.448951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.279468] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7581.666431] LustreError: 180335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.682214] LustreError: 180335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7584.372762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.749184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.931546] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7598.268801] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7604.913278] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.672694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.887167] LustreError: 182150:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7612.907075] Lustre: 182150:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7613.121313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.869442] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7627.161935] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.594071] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7629.599081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.609130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.819976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7630.825493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.849793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.915489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.922504] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.924490] Lustre: Skipped 1 previous similar message [ 7635.831737] Lustre: server umount lustre-MDT0000 complete [ 7639.824971] LustreError: 180316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051993 with bad export cookie 5335690841984140926 [ 7639.845300] LustreError: 181063:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ab2ea99c x1872811136152448/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7639.850194] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.037449] LustreError: 180331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.044209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.094741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7640.110156] Lustre: Skipped 1 previous similar message [ 7640.470462] Lustre: server umount lustre-MDT0001 complete [ 7647.994920] Lustre: server umount lustre-OST0000 complete [ 7660.725059] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 7671.097993] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7705.559990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.745844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7705.764443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7705.816654] Lustre: lustre-MDT0000: new disk, initializing [ 7705.887175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7705.910518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7710.096644] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7720.685035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.828786] Lustre: 185674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7720.908985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7720.915519] Lustre: Skipped 1 previous similar message [ 7721.030789] Lustre: lustre-MDT0001: new disk, initializing [ 7721.182438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7721.233691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7721.243081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7725.464101] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7734.805973] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7740.868621] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7746.956128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.274136] Lustre: lustre-OST0000: new disk, initializing [ 7747.279706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7747.290662] Lustre: 187641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7747.374917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.692914] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7754.291303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7754.304807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7754.387912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7760.903211] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7764.792245] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7765.049780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7769.025201] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7769.233137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7774.704038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.728676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7777.035817] Lustre: server umount lustre-OST0000 complete [ 7782.881190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7782.887466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.900795] Lustre: Skipped 1 previous similar message [ 7782.906677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7782.912233] Lustre: Skipped 1 previous similar message [ 7786.838323] Lustre: server umount lustre-MDT0000 complete [ 7789.983609] LustreError: 185666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052143 with bad export cookie 5335690841984141647 [ 7789.994370] LustreError: 185666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7789.996918] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7790.285993] Lustre: server umount lustre-MDT0001 complete [ 7799.705065] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:35:51 (1786052151) [ 7817.764880] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7820.326434] Key type lgssc unregistered [ 7820.574325] LNet: 190589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.579538] LNetError: 190589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.590958] LNet: Removed LNI 192.168.206.102@tcp [ 7821.335184] Key type .llcrypt unregistered [ 7821.338450] Key type ._llcrypt unregistered [ 7838.386363] Key type ._llcrypt registered [ 7838.387956] Key type .llcrypt registered [ 7838.474837] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7839.512228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7839.614197] alg: No test for adler32 (adler32-zlib) [ 7840.752702] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 7840.985537] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 7842.664152] Key type lgssc registered [ 7843.717678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7851.302767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7851.319687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7852.877703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7857.025199] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7862.244368] LustreError: 191809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7862.272864] LustreError: 191809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7864.335020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.726515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7869.815071] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7877.520867] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7882.670880] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7890.360051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7890.600130] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7890.604976] LustreError: 193676:0:(llog.c:1696:llog_backup()) MGC192.168.206.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7890.616361] Lustre: 193676:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7890.762740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.853487] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7902.973925] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.771830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7905.780060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.801688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7908.325113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7908.329643] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.417083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7909.434044] Lustre: Skipped 4 previous similar messages [ 7911.421664] Lustre: server umount lustre-MDT0000 complete [ 7914.469399] LustreError: 192538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.502572] LustreError: 192538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7916.361983] LustreError: 191791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052270 with bad export cookie 5759661644113835553 [ 7916.368251] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7916.381023] LustreError: 191791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7917.664304] LustreError: 192540:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000618e5f04 x1872811465404160/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7918.257913] Lustre: server umount lustre-MDT0001 complete [ 7933.172624] Lustre: server umount lustre-OST0000 complete [ 7942.381704] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 7952.457429] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7991.318874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7991.520959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7991.607360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7991.702786] Lustre: lustre-MDT0000: new disk, initializing [ 7991.819163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.841283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7996.287504] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8006.132784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.251806] Lustre: 197200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8006.290453] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8006.293258] Lustre: Skipped 1 previous similar message [ 8006.297756] LustreError: 197982:0:(llog.c:1696:llog_backup()) MGC192.168.206.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8006.302827] LustreError: 197982:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8006.306844] Lustre: 197982:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8006.319277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8006.322756] Lustre: Skipped 1 previous similar message [ 8006.386434] Lustre: lustre-MDT0001: new disk, initializing [ 8006.494492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.535081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8006.549079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8011.074629] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8021.165627] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8027.523305] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8034.850599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.234232] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8035.236833] Lustre: Skipped 1 previous similar message [ 8035.243952] LustreError: 199143:0:(llog.c:1696:llog_backup()) MGC192.168.206.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8035.253167] LustreError: 199143:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8035.261388] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8035.275467] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8035.324438] Lustre: lustre-OST0000: new disk, initializing [ 8035.333753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8035.345347] Lustre: 199166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8035.541070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.142842] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8045.190539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8045.213086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8045.378724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8054.001657] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8059.856915] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8060.104947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8064.758782] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8065.039584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8071.142192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8071.149271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.151333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8071.170531] Lustre: Skipped 2 previous similar messages [ 8076.265014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8076.278684] Lustre: Skipped 2 previous similar messages [ 8081.376141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8081.467985] Lustre: server umount lustre-OST0000 complete [ 8086.500112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.511374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8086.521383] Lustre: Skipped 1 previous similar message [ 8086.525160] Lustre: Skipped 2 previous similar messages [ 8099.297458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8099.438291] Lustre: server umount lustre-MDT0000 complete [ 8101.860517] LustreError: 197212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.024668] LustreError: 197192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052457 with bad export cookie 5759661644113836274 [ 8104.035592] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8104.810387] Lustre: server umount lustre-MDT0001 complete [ 8114.532791] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:41:06 (1786052466) [ 8129.952396] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8139.835095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.209827] LustreError: 202296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.220911] LustreError: 202296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8140.257739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.912231] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8145.379721] LustreError: 202297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.572534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.620939] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8157.069928] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8162.928040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.254922] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8178.044752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8178.213985] Lustre: lustre-OST0001: new disk, initializing [ 8178.217099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8178.221531] Lustre: 204605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8178.313058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8178.319316] Lustre: Skipped 2 previous similar messages [ 8184.347995] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8184.419753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8184.435986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8184.524738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8194.597719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8201.740344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8203.452679] Lustre: DEBUG MARKER: ADD OST3 [ 8208.144213] loop: module loaded [ 8208.230615] loop0: detected capacity change from 0 to 204800000 [ 8208.595228] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8208.613936] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8208.628696] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8214.961681] loop0: detected capacity change from 0 to 204800000 [ 8215.670728] Lustre: DEBUG MARKER: STOP OST3 [ 8221.611126] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.838460] Lustre: lustre-OST0003: new disk, initializing [ 8221.845652] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8221.853694] Lustre: 206615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8223.401127] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8223.471208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8223.491205] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8223.531721] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8225.402606] Lustre: Failing over lustre-OST0003 [ 8225.440080] LustreError: 206769:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8225.560901] Lustre: server umount lustre-OST0003 complete [ 8228.076335] LustreError: 203783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.091949] LustreError: 203783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8228.730499] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8228.846106] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.110980] Lustre: DEBUG MARKER: START OST3 again [ 8244.082980] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.197653] LustreError: 203784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.224206] LustreError: 203784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8244.537873] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8244.546207] Lustre: Skipped 1 previous similar message [ 8244.558985] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8245.739072] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8245.825422] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8245.825826] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8250.203652] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8271.230233] Lustre: Failing over lustre-OST0003 [ 8271.424826] Lustre: server umount lustre-OST0003 complete [ 8271.847055] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8271.850350] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.853330] LustreError: Skipped 2 previous similar messages [ 8271.879178] Lustre: Skipped 2 previous similar messages [ 8280.563295] LustreError: 203783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.601832] LustreError: 203783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8290.789034] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.795978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.809717] Lustre: Skipped 2 previous similar messages [ 8290.822558] Lustre: Skipped 8 previous similar messages [ 8296.035269] Lustre: server umount lustre-MDT0000 complete [ 8299.820543] LustreError: 202277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052653 with bad export cookie 5759661644113837156 [ 8299.834840] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.839614] LustreError: 202277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8314.341207] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8314.660931] Lustre: server umount lustre-MDT0001 complete [ 8318.961168] Lustre: server umount lustre-OST0000 complete [ 8322.595126] Lustre: server umount lustre-OST0001 complete [ 8328.422691] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8336.319943] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8373.884312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.180884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8374.250954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8374.346486] Lustre: lustre-MDT0000: new disk, initializing [ 8374.459752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.496753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8380.397093] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8393.088349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.511973] Lustre: 212057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8393.673483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8393.687378] Lustre: Skipped 1 previous similar message [ 8393.877608] Lustre: lustre-MDT0001: new disk, initializing [ 8394.257594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8394.290334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8400.367787] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8410.094631] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8415.526319] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8421.809246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.131340] Lustre: 214028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8423.406273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8423.567149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.460749] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8438.851985] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8444.133355] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8444.376490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8449.217152] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8449.466919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8454.117058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.125132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.131470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8454.131478] Lustre: Skipped 12 previous similar messages [ 8454.164658] Lustre: Skipped 4 previous similar messages [ 8457.481475] Lustre: server umount lustre-OST0000 complete [ 8467.338286] Lustre: server umount lustre-MDT0000 complete [ 8469.476168] LustreError: 212066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8469.505256] LustreError: 212066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8471.200373] LustreError: 212052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052825 with bad export cookie 5759661644113839452 [ 8471.201611] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8471.231815] LustreError: 212052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8471.610364] Lustre: server umount lustre-MDT0001 complete [ 8474.800291] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:47:06 (1786052826) [ 8491.805401] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8501.447431] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8538.846095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.096554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8539.132253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8539.137550] Lustre: Skipped 1 previous similar message [ 8539.217374] Lustre: lustre-MDT0000: new disk, initializing [ 8539.219749] Lustre: Skipped 1 previous similar message [ 8539.305902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8539.314228] Lustre: Skipped 1 previous similar message [ 8543.980681] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8554.052514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.141805] Lustre: 218639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8554.463189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8559.384087] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8569.288651] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.151436] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8581.976354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.322242] Lustre: 220608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8584.249152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8589.738539] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8599.062207] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8603.851758] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8604.105306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8608.670093] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8608.913241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8614.886064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.891603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8614.896723] Lustre: Skipped 2 previous similar messages [ 8614.903561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8614.915995] LustreError: Skipped 1 previous similar message [ 8614.921333] Lustre: Skipped 3 previous similar messages [ 8625.120568] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8625.413577] Lustre: server umount lustre-OST0000 complete [ 8631.272273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8636.168718] Lustre: server umount lustre-MDT0000 complete [ 8638.433562] LustreError: 218650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.442164] LustreError: 218650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8640.287786] LustreError: 218632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786052994 with bad export cookie 5759661644113840334 [ 8640.293156] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.306632] LustreError: 218632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8640.706273] Lustre: server umount lustre-MDT0001 complete [ 8649.425597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.956692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8649.961818] Lustre: Skipped 5 previous similar messages [ 8654.817342] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8663.279514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.253798] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8674.908142] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8679.902465] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8686.479284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8694.061406] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8701.234910] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8714.723153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.756472] Lustre: server umount lustre-OST0000 complete [ 8737.249748] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.388618] Lustre: server umount lustre-MDT0000 complete [ 8741.382490] LustreError: 222810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786053095 with bad export cookie 5759661644113841216 [ 8741.393662] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.411036] LustreError: 222810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8741.924547] Lustre: server umount lustre-MDT0001 complete [ 8839.500443] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 8842.432142] Key type lgssc unregistered [ 8842.868746] LNet: 226506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8842.878780] LNetError: 226506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8842.908236] LNet: Removed LNI 192.168.206.102@tcp [ 8843.834424] Key type .llcrypt unregistered [ 8843.837463] Key type ._llcrypt unregistered [ 8867.534480] Key type ._llcrypt registered [ 8867.536635] Key type .llcrypt registered [ 8867.660876] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8882.290773] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8883.319693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8883.453088] alg: No test for adler32 (adler32-zlib) [ 8884.738518] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 8885.024695] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 8886.688257] Key type lgssc registered [ 8887.747306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8923.577364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8923.595520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.947574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8924.981151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8925.067581] Lustre: lustre-MDT0000: new disk, initializing [ 8925.188964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8925.204522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8930.038676] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8941.426887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.609825] Lustre: 229859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8941.687377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8941.691057] Lustre: Skipped 1 previous similar message [ 8941.826704] Lustre: lustre-MDT0001: new disk, initializing [ 8941.944476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8941.984280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8942.001432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8946.934124] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8956.741957] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8963.432049] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8971.645653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8971.972962] Lustre: lustre-OST0000: new disk, initializing [ 8971.977991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8971.987361] Lustre: 231827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8972.082581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8979.125064] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8979.537943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8979.545327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8979.689215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8989.142378] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8993.542564] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8993.894545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8998.637095] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8998.832257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9005.537982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9005.542497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.545047] LustreError: Skipped 1 previous similar message [ 9005.545615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9005.629775] Lustre: Skipped 1 previous similar message [ 9006.851846] Lustre: server umount lustre-OST0000 complete [ 9010.664222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.685261] Lustre: Skipped 1 previous similar message [ 9010.692043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9010.705460] Lustre: Skipped 2 previous similar messages [ 9015.780855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.804030] Lustre: Skipped 1 previous similar message [ 9017.392647] Lustre: server umount lustre-MDT0000 complete [ 9021.298161] LustreError: 229851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786053375 with bad export cookie 3153901879384928283 [ 9021.304708] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.309677] LustreError: 229851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9021.676164] Lustre: server umount lustre-MDT0001 complete [ 9039.518738] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9049.270488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.621238] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.639560] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9049.667688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.719301] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9054.689530] LustreError: 234761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9059.811986] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.008332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.234231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9064.845718] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9067.178021] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9072.614229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9072.863174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.252664] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9095.065644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9095.362860] Lustre: lustre-OST0001: new disk, initializing [ 9095.380164] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9095.394263] Lustre: 237068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9095.632298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9103.273362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9103.309228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9103.483808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9109.191374] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9128.755374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9142.891057] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9165.970872] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:58:37 (1786053517) [ 9168.865336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9168.867853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.884163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.876778] Lustre: server umount lustre-MDT0000 complete [ 9175.016536] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.037279] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9178.168602] LustreError: 234741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786053532 with bad export cookie 3153901879384929165 [ 9178.172035] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.181789] LustreError: 234741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9178.568353] Lustre: server umount lustre-MDT0001 complete [ 9191.350763] Lustre: server umount lustre-OST0000 complete [ 9206.908446] Lustre: server umount lustre-OST0001 complete [ 9213.532969] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 9223.179541] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9258.887593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.102484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9259.164069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9259.284985] Lustre: lustre-MDT0000: new disk, initializing [ 9259.356314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.371139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9264.700049] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9275.227567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.367467] Lustre: 241044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9275.424702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9275.429918] Lustre: Skipped 1 previous similar message [ 9275.523587] Lustre: lustre-MDT0001: new disk, initializing [ 9275.678841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9275.686698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9280.533275] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9289.503885] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9295.073357] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9301.684646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.967100] Lustre: lustre-OST0000: new disk, initializing [ 9301.972043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9301.977394] Lustre: 243012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9303.399151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9303.410432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9303.495483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9308.093606] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9316.055585] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9320.452765] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9320.631899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.795095] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9324.993995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9329.122162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9329.123181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.139639] LustreError: Skipped 1 previous similar message [ 9329.148310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9329.148319] Lustre: Skipped 4 previous similar messages [ 9329.182020] Lustre: Skipped 4 previous similar messages [ 9333.050061] Lustre: server umount lustre-OST0000 complete [ 9337.313594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9337.333921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.359153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.368123] Lustre: Skipped 1 previous similar message [ 9351.136617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9351.372263] Lustre: server umount lustre-MDT0000 complete [ 9354.722893] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.762552] LustreError: 241051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9355.300447] LustreError: 243011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786053709 with bad export cookie 3153901879384931300 [ 9355.308663] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9355.314926] LustreError: 243011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9355.701307] Lustre: server umount lustre-MDT0001 complete [ 9364.883886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.408343] LustreError: 245238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9365.533949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.544815] Lustre: Skipped 2 previous similar messages [ 9370.225919] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9378.417880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.460989] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9391.552218] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9398.195976] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9406.928785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9415.553278] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9422.893526] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9567.201209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9567.214389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.226093] Lustre: Skipped 1 previous similar message [ 9567.230665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9567.237360] Lustre: Skipped 6 previous similar messages [ 9573.326930] Lustre: server umount lustre-OST0000 complete [ 9578.467041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.480934] LustreError: Skipped 1 previous similar message [ 9582.992979] Lustre: server umount lustre-MDT0000 complete [ 9585.645222] LustreError: 246450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.680744] LustreError: 246450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9586.864378] LustreError: 245218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786053940 with bad export cookie 3153901879384932182 [ 9586.869396] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9586.876090] LustreError: 245218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9587.201847] Lustre: server umount lustre-MDT0001 complete [ 9690.888255] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 9693.627041] Key type lgssc unregistered [ 9693.930551] LNet: 248852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9693.935600] LNetError: 248852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9693.955927] LNet: Removed LNI 192.168.206.102@tcp [ 9694.986161] Key type .llcrypt unregistered [ 9694.996271] Key type ._llcrypt unregistered [ 9722.141561] Key type ._llcrypt registered [ 9722.149508] Key type .llcrypt registered [ 9722.304680] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 9737.914327] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9739.403630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9739.426768] alg: No test for adler32 (adler32-zlib) [ 9740.543135] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [ 9740.823495] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 9742.552164] Key type lgssc registered [ 9743.572443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9790.649935] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9801.161707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9801.178254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9802.517377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9802.600682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9802.709451] Lustre: lustre-MDT0000: new disk, initializing [ 9802.805328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9802.821599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9807.803235] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9817.948933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.042351] Lustre: 252922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9818.084414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9818.087249] Lustre: Skipped 1 previous similar message [ 9818.152113] Lustre: lustre-MDT0001: new disk, initializing [ 9818.249641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9818.314970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9818.321534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9822.365657] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9827.221352] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9834.261683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9834.501759] Lustre: lustre-OST0000: new disk, initializing [ 9834.508085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9834.514924] Lustre: 254554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9834.600133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9838.439805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9838.459498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9838.532238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9840.912427] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9851.807026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9852.020720] Lustre: lustre-OST0001: new disk, initializing [ 9852.035078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9852.048632] Lustre: 255423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9852.176937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9859.184389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9859.201102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9859.318609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9859.480658] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9869.531437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9880.810689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9895.393230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9895.394804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.407409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9895.438790] Lustre: Skipped 3 previous similar messages [ 9899.381823] Lustre: server umount lustre-MDT0000 complete [ 9903.441320] LustreError: 252914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786054257 with bad export cookie 5435211923292632301 [ 9903.447300] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9903.465734] LustreError: 252914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9903.797100] Lustre: server umount lustre-MDT0001 complete [ 9919.217616] Lustre: server umount lustre-OST0000 complete [ 9921.504139] Lustre: 250430:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786054259/real 1786054259] req@ffff987a0afc2680 x1872813457747968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786054275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9921.529471] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9923.513444] Lustre: server umount lustre-OST0001 complete [ 9938.286330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9948.845726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9957.904657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9967.907134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9990.236232] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10002.062991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.095234] Lustre: lustre-MDT0000: reset Object Index mappings [10002.572456] LustreError: 260434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.588534] LustreError: 260434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10002.663874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.915608] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10008.034613] LustreError: 260435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.138726] LustreError: 260434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.321372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.359752] Lustre: lustre-MDT0001: reset Object Index mappings [10016.041240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10020.209712] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10023.363971] Lustre: 261577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10029.929809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10036.520288] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10040.491341] LustreError: 262478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.817265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10045.039771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10045.043149] Lustre: Skipped 1 previous similar message [10048.130478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10048.153758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10050.848948] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10059.165442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10063.442474] Lustre: 263448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10068.967619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.972229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.980431] Lustre: Skipped 1 previous similar message [10068.996976] Lustre: Skipped 6 previous similar messages [10070.809301] Lustre: server umount lustre-MDT0000 complete [10074.088672] LustreError: 260980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.102518] LustreError: 260980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10074.744751] LustreError: 261955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786054428 with bad export cookie 5435211923292639091 [10074.760654] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.135221] Lustre: server umount lustre-MDT0001 complete [10088.570822] Lustre: server umount lustre-OST0000 complete [10101.452700] Lustre: server umount lustre-OST0001 complete [10120.376803] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:14:32 (1786054472) [10136.610778] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [10145.911934] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10180.433563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.784365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10180.841563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10181.001992] Lustre: lustre-MDT0000: new disk, initializing [10181.110063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.148511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10185.540711] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10196.415172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.584423] Lustre: 267702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10196.644686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10196.657509] Lustre: Skipped 1 previous similar message [10196.766752] Lustre: lustre-MDT0001: new disk, initializing [10196.901717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10196.916816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10201.225754] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10209.975573] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10215.695861] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10222.243420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10222.650576] Lustre: lustre-OST0000: new disk, initializing [10222.654462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10222.660251] Lustre: 269669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10224.012873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10224.024433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10224.109264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10229.244588] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10238.482719] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10243.708839] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10243.999683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10249.263475] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10249.481510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10254.825833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10254.829726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.849486] Lustre: Skipped 2 previous similar messages [10254.855056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10254.859663] Lustre: Skipped 1 previous similar message [10259.959512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10265.072349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10265.081273] Lustre: Skipped 2 previous similar messages [10266.083206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10266.244819] Lustre: server umount lustre-OST0000 complete [10270.178801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.203330] Lustre: Skipped 2 previous similar messages [10275.297179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.310402] Lustre: Skipped 3 previous similar messages [10276.513545] Lustre: server umount lustre-MDT0000 complete [10280.245931] LustreError: 268497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786054634 with bad export cookie 5435211923292641184 [10280.247780] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.255396] LustreError: 268497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10280.630416] Lustre: server umount lustre-MDT0001 complete [10298.488542] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10308.732588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.171917] LustreError: 272606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.235357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10309.239320] Lustre: Skipped 2 previous similar messages [10313.369328] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10321.184202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.334191] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10328.163733] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10334.300153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.854189] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10348.790784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10348.970216] Lustre: lustre-OST0001: new disk, initializing [10348.978620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10348.986187] Lustre: 274912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10350.869651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10350.886539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10351.034538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10355.690916] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10364.684460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10368.011359] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10374.016037] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10374.270651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10381.799090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.803164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10381.811145] Lustre: Skipped 2 previous similar messages [10381.818405] Lustre: Skipped 1 previous similar message [10395.104172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10395.220443] Lustre: server umount lustre-MDT0000 complete [10397.158967] LustreError: 272607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.170023] LustreError: 272607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10398.111664] LustreError: 275676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786054751 with bad export cookie 5435211923292642066 [10398.116947] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10398.121537] LustreError: 275676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10398.377957] Lustre: server umount lustre-MDT0001 complete [10411.649250] Lustre: server umount lustre-OST0000 complete [10425.923356] Lustre: server umount lustre-OST0001 complete [10441.852596] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10451.919268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.366212] LustreError: 277971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.442740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10452.446774] Lustre: Skipped 3 previous similar messages [10456.593838] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10464.564688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.068869] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10472.458569] Lustre: 279112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10480.206828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.950836] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10495.972409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.353697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10497.367248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10502.881685] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10510.869247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10514.712206] Lustre: 280990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10519.738498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10519.855147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10527.913820] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:21:20 (1786054880) [10543.591251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.592182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10543.606558] Lustre: Skipped 4 previous similar messages [10543.610357] Lustre: Skipped 12 previous similar messages [10556.384122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10558.536184] Lustre: server umount lustre-MDT0000 complete [10558.947478] LustreError: 277968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.962158] LustreError: 277968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10561.341110] LustreError: 277953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786054915 with bad export cookie 5435211923292648604 [10561.359830] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10561.360900] LustreError: 277953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10567.729379] Lustre: server umount lustre-MDT0001 complete [10577.479261] Lustre: server umount lustre-OST0000 complete [10586.895975] Lustre: server umount lustre-OST0001 complete [10592.984274] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [10601.396443] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10642.920614] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10651.502320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.727543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10651.761730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10651.839341] Lustre: lustre-MDT0000: new disk, initializing [10651.914182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10655.278670] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10663.541699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.616639] Lustre: 285232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10663.842058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10666.893218] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10676.132957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.380758] Lustre: 286862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10677.777465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10681.651051] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10691.492280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10691.645150] Lustre: 287732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10692.965257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10696.979898] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10705.931170] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10709.081316] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10709.084456] Lustre: Skipped 1 previous similar message [10724.197789] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:24:36 (1786055076) [10728.933187] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.937456] Lustre: Skipped 4 previous similar messages [10728.944106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10728.953782] Lustre: Skipped 15 previous similar messages [10729.041647] Lustre: server umount lustre-MDT0000 complete [10732.620623] LustreError: 286025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786055086 with bad export cookie 5435211923292656241 [10732.628635] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10732.633577] LustreError: 286025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10732.953204] Lustre: server umount lustre-MDT0001 complete [10746.405914] Lustre: server umount lustre-OST0000 complete [10749.222929] Lustre: 250427:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786055087/real 1786055087] req@ffff987a01683b80 x1872813458052608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786055103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10749.719129] Lustre: server umount lustre-OST0001 complete [10757.639051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.113454] LustreError: 289977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10758.137238] LustreError: 289977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10758.228374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10758.233020] Lustre: Skipped 7 previous similar messages [10761.888252] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10769.282156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.416706] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10779.076408] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10783.525395] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10788.878029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10793.671684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10799.352416] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10816.753358] loop1: detected capacity change from 0 to 204800000 [10816.916864] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10816.924933] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10816.933546] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10821.079032] loop1: detected capacity change from 0 to 204800000 [10825.875642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10825.969666] LustreError: 289964:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3411: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10825.978662] LustreError: 289964:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3411 [10825.983280] LustreError: 289964:0:(mgs_handler.c:517:mgs_target_reg()) fs3411-MDT0001: Error checking target index: rc = -13 [10825.993639] LustreError: fs3411-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10826.002268] LustreError: 293692:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10826.007626] LustreError: 293692:0:(tgt_mount.c:2129:server_put_super()) no obd fs3411-MDT0001 [10826.011697] LustreError: 293692:0:(tgt_mount.c:116:server_deregister_mount()) fs3411-MDT0001 not registered [10826.037826] Lustre: server umount fs3411-MDT0001 complete [10826.040302] LustreError: 293692:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10832.781086] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.865918] Lustre: Setting parameter fs3411-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3411-MDT0001 [10832.916645] Lustre: srv-fs3411-MDT0001: No data found on store. Initialize space. [10832.920691] Lustre: Skipped 4 previous similar messages [10833.029308] Lustre: fs3411-MDT0001: new disk, initializing [10833.031917] Lustre: Skipped 3 previous similar messages [10833.120337] Lustre: srv-fs3411-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10833.132969] Lustre: 294158:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3411-MDT0001-mdtlov: can't create new llog: rc = -115 [10833.148543] LustreError: 294158:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3411-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10836.521834] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10841.979168] Lustre: server umount fs3411-MDT0001 complete [10852.437969] loop1: detected capacity change from 0 to 204800000 [10856.796196] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.090056] Lustre: 295726:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3411-MDT0001-mdtlov: can't create new llog: rc = -115 [10857.101323] LustreError: 295726:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3411-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10861.651980] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10865.531175] Lustre: server umount fs3411-MDT0001 complete [10875.365671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.368896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10875.377538] Lustre: Skipped 3 previous similar messages [10875.390514] Lustre: Skipped 2 previous similar messages [10879.615503] Lustre: server umount lustre-MDT0000 complete [10880.494049] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10880.509872] LustreError: Skipped 1 previous similar message [10887.965105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.200453] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10893.359109] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10893.813721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10900.585340] loop1: detected capacity change from 0 to 204800000 [10905.512496] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.784761] Lustre: 298172:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3411-MDT0001-mdtlov: can't create new llog: rc = -115 [10905.792587] LustreError: 298172:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3411-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10909.131687] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10913.723319] Lustre: server umount fs3411-MDT0001 complete [10934.766057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10940.999437] Lustre: server umount lustre-OST0000 complete [10951.754535] Lustre: server umount lustre-MDT0000 complete [10956.758510] LustreError: 289958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786055310 with bad export cookie 5435211923292658922 [10956.768471] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.773776] LustreError: 289958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10957.239960] Lustre: server umount lustre-MDT0001 complete [10967.466451] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [10969.996292] Key type lgssc unregistered [10970.265707] LNet: 300519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10970.274824] LNetError: 300519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10970.299796] LNet: Removed LNI 192.168.206.102@tcp [10971.226395] Key type .llcrypt unregistered [10971.229935] Key type ._llcrypt unregistered [10989.355644] Key type ._llcrypt registered [10989.359228] Key type .llcrypt registered [10989.480242] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10990.674973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10990.909555] alg: No test for adler32 (adler32-zlib) [10991.904520] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [10992.116095] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [10993.816235] Key type lgssc registered [10994.828608] Lustre: Echo OBD driver; http://www.lustre.org/ [11001.988805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11002.014385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.451444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11007.802831] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11013.608349] LustreError: 301791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11013.633040] LustreError: 301791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11015.332336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.694258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11019.081940] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11025.355134] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11030.063925] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11036.271399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11036.640127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11040.749346] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.281141] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11045.870279] LustreError: 303621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.559221] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11048.943049] LustreError: 303621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.963612] LustreError: 303621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11054.063248] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.089642] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11064.294600] LustreError: 303624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.317938] LustreError: 303624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11065.390609] loop1: detected capacity change from 0 to 204800000 [11065.672833] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11065.682528] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11065.693159] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11071.313551] loop1: detected capacity change from 0 to 204800000 [11076.922107] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.106281] Lustre: fs21624-OST0000: new disk, initializing [11077.118647] Lustre: srv-fs21624-OST0000: No data found on store. Initialize space. [11077.135536] Lustre: 305525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11077.226243] Lustre: fs21624-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11081.931412] LustreError: 303892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.956385] LustreError: 303892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11084.060991] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11093.020814] Lustre: server umount fs21624-OST0000 complete [11104.621332] loop1: detected capacity change from 0 to 204800000 [11110.669022] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11111.020803] Lustre: fs21624-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11115.511582] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.531163] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11118.020607] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11125.746135] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11127.022647] loop2: detected capacity change from 0 to 204800000 [11127.198897] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11127.209289] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11127.223136] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11130.854465] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11132.206973] loop2: detected capacity change from 0 to 204800000 [11137.100653] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.147520] LustreError: fs21624-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11137.154825] LustreError: fs21624-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11137.161888] LustreError: 308565:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11137.167446] LustreError: 308565:0:(tgt_mount.c:2129:server_put_super()) no obd fs21624-OST0001 [11137.172229] LustreError: 308565:0:(tgt_mount.c:116:server_deregister_mount()) fs21624-OST0001 not registered [11137.200701] Lustre: server umount fs21624-OST0001 complete [11137.204215] LustreError: 308565:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11147.393271] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.486954] Lustre: fs21624-OST0001: new disk, initializing [11147.490831] Lustre: srv-fs21624-OST0001: No data found on store. Initialize space. [11147.498833] Lustre: 309116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11147.570508] Lustre: fs21624-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11154.307766] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11163.064731] Lustre: server umount fs21624-OST0000 complete [11172.866458] Lustre: server umount fs21624-OST0001 complete [11182.062500] LustreError: 303621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.086698] LustreError: 303621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11185.713718] loop1: detected capacity change from 0 to 204800000 [11192.073235] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11192.332610] Lustre: fs21624-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11198.794449] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11204.984301] loop2: detected capacity change from 0 to 204800000 [11210.361997] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.194427] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11230.191564] Lustre: server umount fs21624-OST0000 complete [11240.338768] Lustre: server umount fs21624-OST0001 complete [11253.738219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11253.742459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.746196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11253.764335] Lustre: Skipped 1 previous similar message [11258.540908] Lustre: server umount lustre-OST0000 complete [11263.973857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.986098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11263.990488] Lustre: Skipped 1 previous similar message [11264.012147] Lustre: Skipped 2 previous similar messages [11269.092926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.101966] Lustre: Skipped 1 previous similar message [11273.191029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11273.198113] Lustre: Skipped 1 previous similar message [11276.256116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11276.470966] Lustre: server umount lustre-MDT0000 complete [11280.289074] LustreError: 301771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786055634 with bad export cookie 9624027653905928315 [11280.295285] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11280.305326] LustreError: 301771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11280.530640] Lustre: server umount lustre-MDT0001 complete [11291.734317] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11294.091760] Key type lgssc unregistered [11294.397217] LNet: 315045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11294.407558] LNetError: 315045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11294.435442] LNet: Removed LNI 192.168.206.102@tcp [11295.239199] Key type .llcrypt unregistered [11295.241356] Key type ._llcrypt unregistered [11330.780944] Key type ._llcrypt registered [11330.783890] Key type .llcrypt registered [11330.886770] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [11346.227869] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11347.545692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11347.590175] alg: No test for adler32 (adler32-zlib) [11348.587186] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [11348.822042] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11350.464163] Key type lgssc registered [11351.604477] Lustre: Echo OBD driver; http://www.lustre.org/ [11390.160855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11390.192044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11391.534701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11391.613515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11391.707384] Lustre: lustre-MDT0000: new disk, initializing [11391.825497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11391.848521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11396.363155] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11406.261066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.423497] Lustre: 318909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11406.489440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11406.498352] Lustre: Skipped 1 previous similar message [11406.591237] Lustre: lustre-MDT0001: new disk, initializing [11406.730559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11406.778109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11406.792453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11411.560554] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11419.488124] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11424.449491] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11430.487343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11430.720284] Lustre: lustre-OST0000: new disk, initializing [11430.725978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11430.733308] Lustre: 320876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11430.803278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11436.181178] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11438.657057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11438.669311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11438.748197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11446.308226] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11448.457071] LustreError: 317081:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11448.498896] Lustre: Mounted lustre-client - version 2.17.56_53_g253671b [11449.527827] Lustre: Unmounted lustre-client [11459.558192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11459.567460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11459.573195] LustreError: Skipped 1 previous similar message [11459.592055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11459.621548] Lustre: Skipped 1 previous similar message [11464.172866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11464.190290] Lustre: Skipped 1 previous similar message [11465.556850] Lustre: server umount lustre-OST0000 complete [11473.381422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11473.394275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11473.417261] Lustre: Skipped 1 previous similar message [11473.430070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11474.402111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.743751] Lustre: server umount lustre-MDT0000 complete [11479.602286] LustreError: 318901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786055833 with bad export cookie 11246388690628508204 [11479.610099] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11479.611050] LustreError: 318901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11480.010287] Lustre: server umount lustre-MDT0001 complete [11489.626130] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11492.293926] Key type lgssc unregistered [11504.480565] LNet: 1 local NIs in recovery (showing 1): 192.168.206.102@tcp [11504.865640] LNet: 323042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11504.900828] LNetError: 323042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11580.711729] LNet: Removed LNI 192.168.206.102@tcp [11581.556597] Key type .llcrypt unregistered [11581.558879] Key type ._llcrypt unregistered [11599.970092] Key type ._llcrypt registered [11599.973633] Key type .llcrypt registered [11600.086731] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [11613.996445] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11614.635252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11614.712387] alg: No test for adler32 (adler32-zlib) [11615.698582] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [11615.907978] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11617.592156] Key type lgssc registered [11618.509469] Lustre: Echo OBD driver; http://www.lustre.org/ [11651.970978] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:40:03 (1786056003) [11659.543428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11659.577950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11660.748338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11660.773714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11660.837977] Lustre: lustre-MDT0000: new disk, initializing [11660.900352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11660.913243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11665.400951] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11675.391366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.527686] Lustre: 326271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11675.614992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11675.619291] Lustre: Skipped 1 previous similar message [11675.711772] Lustre: lustre-MDT0001: new disk, initializing [11675.792238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11675.831995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11675.841880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11680.916779] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11689.722454] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11695.397992] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11701.356486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.576571] Lustre: lustre-OST0000: new disk, initializing [11701.583691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11701.594986] Lustre: 328239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11701.672446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11706.752944] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11712.077178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11712.086096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11712.202587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11714.735503] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11732.961233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11732.967139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11732.970484] LustreError: Skipped 1 previous similar message [11732.971598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11732.992605] Lustre: Skipped 1 previous similar message [11734.688362] Lustre: server umount lustre-OST0000 complete [11738.089598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11738.101570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11738.107714] Lustre: Skipped 1 previous similar message [11738.132381] Lustre: Skipped 2 previous similar messages [11743.201900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11743.211957] Lustre: Skipped 1 previous similar message [11744.381764] Lustre: server umount lustre-MDT0000 complete [11747.471183] LustreError: 328238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786056101 with bad export cookie 5671283215790214393 [11747.475374] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11747.479230] LustreError: 328238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11747.695267] Lustre: server umount lustre-MDT0001 complete [11756.568758] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11759.003356] Key type lgssc unregistered [11759.259761] LNet: 330148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11759.267636] LNetError: 330148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11759.285898] LNet: Removed LNI 192.168.206.102@tcp [11759.984712] Key type .llcrypt unregistered [11759.987228] Key type ._llcrypt unregistered [11777.082908] Key type ._llcrypt registered [11777.085274] Key type .llcrypt registered [11777.172231] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11778.082287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11778.185123] alg: No test for adler32 (adler32-zlib) [11779.246166] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [11779.472350] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11781.120175] Key type lgssc registered [11782.011906] Lustre: Echo OBD driver; http://www.lustre.org/ [11788.022785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11788.056751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.543926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.467096] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11800.033621] LustreError: 331401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.048341] LustreError: 331401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11800.616812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.936682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11804.381737] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11809.538360] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11815.079629] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11822.786314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11823.098830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11830.087976] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11837.850158] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11847.131907] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11848.790062] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11852.259454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11852.263281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.275953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.818060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.819172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.824409] Lustre: Skipped 1 previous similar message [11854.839892] Lustre: Skipped 1 previous similar message [11858.217489] Lustre: server umount lustre-MDT0000 complete [11859.938855] LustreError: 331402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.947378] LustreError: 331402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11862.040242] LustreError: 332136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786056215 with bad export cookie 8649246921314319839 [11862.049584] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11862.054907] LustreError: 332136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11865.057456] LustreError: 331397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.057928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11865.088971] LustreError: 331397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11865.117871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11865.125852] Lustre: Skipped 2 previous similar messages [11867.105139] LustreError: 331396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.114715] LustreError: 331396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11868.403163] Lustre: server umount lustre-MDT0001 complete [11871.974734] Lustre: server umount lustre-OST0000 complete [11879.947567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11883.904887] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11886.865593] Lustre: 335580:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.102@tcp [11889.104114] Lustre: server umount MGS complete [11898.509153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.102789] LustreError: 336152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.186952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11904.270074] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11909.611608] LustreError: 336153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.644601] LustreError: 336153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11912.546037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11912.962926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11917.253386] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11926.392456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11926.779575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11932.141447] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11938.616780] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11940.077443] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11944.844758] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11948.396656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11953.641206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11953.650786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11953.672602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11953.681787] Lustre: Skipped 2 previous similar messages [11958.534683] Lustre: server umount lustre-OST0000 complete [11964.388644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11964.396195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.411095] Lustre: Skipped 1 previous similar message [11964.419182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.431508] Lustre: Skipped 2 previous similar messages [11967.694736] Lustre: server umount lustre-MDT0000 complete [11970.020870] LustreError: 336881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.039586] LustreError: 336881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11971.030990] LustreError: 336134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786056324 with bad export cookie 8649246921314320679 [11971.041157] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.052215] LustreError: 336134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11971.369557] Lustre: server umount lustre-MDT0001 complete [11980.223308] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11982.667420] Key type lgssc unregistered [11982.891088] LNet: 339593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11982.896134] LNetError: 339593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11982.908620] LNet: Removed LNI 192.168.206.102@tcp [11983.719570] Key type .llcrypt unregistered [11983.722467] Key type ._llcrypt unregistered [12000.449508] Key type ._llcrypt registered [12000.453170] Key type .llcrypt registered [12000.542686] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [12016.534367] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [12017.889515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12017.929037] alg: No test for adler32 (adler32-zlib) [12019.725657] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [12020.109763] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [12021.936143] Key type lgssc registered [12023.171861] Lustre: Echo OBD driver; http://www.lustre.org/ [12062.965855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12063.013342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12064.254451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12064.308483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12064.419265] Lustre: lustre-MDT0000: new disk, initializing [12064.481737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12064.503461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12069.139943] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12078.416360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.503560] Lustre: 342757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12078.546240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12078.549464] Lustre: Skipped 1 previous similar message [12078.607457] Lustre: lustre-MDT0001: new disk, initializing [12078.673313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12078.701191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12078.721465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12082.492309] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12091.626987] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12096.601217] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12103.511217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.792627] Lustre: lustre-OST0000: new disk, initializing [12103.804482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12103.816410] Lustre: 344723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12103.936682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12109.069434] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12110.382634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12110.394951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12110.491547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12117.488252] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12122.664800] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12122.951746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12128.017887] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12128.375147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12135.913680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12135.939561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.961028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12136.420901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12136.446185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12136.469985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12136.478932] Lustre: Skipped 1 previous similar message [12138.268672] Lustre: server umount lustre-OST0000 complete [12145.121735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12145.129212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.145319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12149.757596] Lustre: server umount lustre-MDT0000 complete [12153.174701] LustreError: 342749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786056507 with bad export cookie 8198023262560499843 [12153.188268] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12153.201614] LustreError: 342749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12153.609589] Lustre: server umount lustre-MDT0001 complete [12161.578810] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:48:33 (1786056513) [12175.067539] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [12184.701771] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [12223.689885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12223.896181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12223.921437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12224.011952] Lustre: lustre-MDT0000: new disk, initializing [12224.094053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12224.115505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12228.393419] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12238.064600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.172734] Lustre: 349481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12238.228518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12238.234327] Lustre: Skipped 1 previous similar message [12238.302705] Lustre: lustre-MDT0001: new disk, initializing [12238.373490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12238.399229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12238.408274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12243.101872] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12254.494909] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12262.100799] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12268.254077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12268.625347] Lustre: lustre-OST0000: new disk, initializing [12268.628511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12268.635990] Lustre: 351452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12268.723974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12274.679164] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12276.777455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12276.791195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12276.884121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12285.683281] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12322.788452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.794239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12322.812504] Lustre: Skipped 2 previous similar messages [12322.821929] Lustre: Skipped 3 previous similar messages [12327.890655] Lustre: server umount lustre-OST0000 complete [12333.029728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.039252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.047671] Lustre: Skipped 1 previous similar message [12337.840661] Lustre: server umount lustre-MDT0000 complete [12338.145502] LustreError: 349485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.161919] LustreError: 349485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12341.130964] LustreError: 349473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786056695 with bad export cookie 8198023262560500725 [12341.134558] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.141132] LustreError: 349473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12341.377374] Lustre: server umount lustre-MDT0001 complete [12351.711522] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12354.201247] Key type lgssc unregistered [12354.413586] LNet: 353627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12354.420982] LNetError: 353627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12354.440655] LNet: Removed LNI 192.168.206.102@tcp [12355.226398] Key type .llcrypt unregistered [12355.229938] Key type ._llcrypt unregistered [12452.333471] Key type ._llcrypt registered [12452.335409] Key type .llcrypt registered [12452.432298] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12452.726306] Key type .llcrypt unregistered [12452.728605] Key type ._llcrypt unregistered [12532.268295] Key type ._llcrypt registered [12532.271327] Key type .llcrypt registered [12532.362393] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12532.753398] Key type .llcrypt unregistered [12532.756490] Key type ._llcrypt unregistered [12569.324606] Key type ._llcrypt registered [12569.327773] Key type .llcrypt registered [12569.487288] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12569.967400] Key type .llcrypt unregistered [12569.968784] Key type ._llcrypt unregistered [12605.875781] Key type ._llcrypt registered [12605.877931] Key type .llcrypt registered [12605.988942] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12606.336352] Key type .llcrypt unregistered [12606.339189] Key type ._llcrypt unregistered [12686.047461] Key type ._llcrypt registered [12686.050817] Key type .llcrypt registered [12686.234073] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12686.498323] Key type .llcrypt unregistered [12686.501780] Key type ._llcrypt unregistered [12722.354670] Key type ._llcrypt registered [12722.357898] Key type .llcrypt registered [12722.504775] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12722.793765] Key type .llcrypt unregistered [12722.796496] Key type ._llcrypt unregistered [12755.246627] Key type ._llcrypt registered [12755.250656] Key type .llcrypt registered [12755.355497] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12755.670928] Key type .llcrypt unregistered [12755.673920] Key type ._llcrypt unregistered [12786.163159] Key type ._llcrypt registered [12786.165220] Key type .llcrypt registered [12786.225375] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12786.470314] Key type .llcrypt unregistered [12786.473809] Key type ._llcrypt unregistered [12821.212627] Key type ._llcrypt registered [12821.215299] Key type .llcrypt registered [12821.335909] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12821.576277] Key type .llcrypt unregistered [12821.577957] Key type ._llcrypt unregistered [12856.033046] Key type ._llcrypt registered [12856.034783] Key type .llcrypt registered [12856.157319] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12856.472278] Key type .llcrypt unregistered [12856.477438] Key type ._llcrypt unregistered [12919.493665] Key type ._llcrypt registered [12919.495278] Key type .llcrypt registered [12919.572758] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12919.842424] Key type .llcrypt unregistered [12919.847735] Key type ._llcrypt unregistered [12949.306506] Key type ._llcrypt registered [12949.309955] Key type .llcrypt registered [12949.390623] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12949.623501] Key type .llcrypt unregistered [12949.625672] Key type ._llcrypt unregistered [12977.980482] Key type ._llcrypt registered [12977.984223] Key type .llcrypt registered [12978.124346] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12978.552289] Key type .llcrypt unregistered [12978.557959] Key type ._llcrypt unregistered [13035.927539] Key type ._llcrypt registered [13035.930752] Key type .llcrypt registered [13036.011582] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [13050.518149] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13052.448412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13052.532195] alg: No test for adler32 (adler32-zlib) [13053.708515] Lustre: Lustre: Build Version: 2.17.56_53_g253671b [13054.000569] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [13055.705705] Key type lgssc registered [13056.757792] Lustre: Echo OBD driver; http://www.lustre.org/ [13087.107573] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13087.131774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13088.325478] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13088.351759] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13088.404644] Lustre: tst250fs-MDT0000: new disk, initializing [13088.453838] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13088.473092] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13092.455754] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13101.953645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.056588] Lustre: 373997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13102.093845] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13102.096850] Lustre: Skipped 1 previous similar message [13102.146725] Lustre: tst250fs-MDT0001: new disk, initializing [13102.211333] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13102.250957] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13102.263223] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13106.208919] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13110.481313] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13116.549486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13116.777654] Lustre: tst250fs-OST0000: new disk, initializing [13116.781685] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13116.788058] Lustre: 375625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13116.867543] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13120.190296] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13120.207403] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13120.378443] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13122.172537] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13131.099224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13131.251499] Lustre: tst250fs-OST0001: new disk, initializing [13131.253806] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13131.257725] Lustre: 376498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13131.343727] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13136.994668] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13139.511842] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13139.519551] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13139.601583] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13141.622675] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13143.811294] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13143.816833] Lustre: Skipped 2 previous similar messages [13158.884993] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13158.896620] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13158.913534] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13159.907036] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13159.908232] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.610498] Lustre: server umount tst250fs-MDT0000 complete [13166.363800] LustreError: 373987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786057520 with bad export cookie 11280027431871739 [13166.367239] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13166.373323] LustreError: 373987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13166.631560] Lustre: server umount tst250fs-MDT0001 complete [13179.417733] Lustre: server umount tst250fs-OST0000 complete [13192.288229] Lustre: server umount tst250fs-OST0001 complete [13204.504807] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [13210.293378] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13233.146848] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:06:25 (1786057585) [13234.538602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13251.791828] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [13261.942318] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13299.339763] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13000 sec ========= 19:07:30 (1786057650) [13301.551748] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:07:32 (1786057652) === [13305.233261] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:07:37 (1786057657) === [13338.308768] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [13341.359036] Key type lgssc unregistered [13341.677554] LNet: 385627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13341.692147] LNetError: 385627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13341.718651] LNet: Removed LNI 192.168.206.102@tcp [13342.677335] Key type .llcrypt unregistered [13342.681460] Key type ._llcrypt unregistered