[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533386663 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003087] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009010] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.013010] LSM: Security Framework initializing [ 0.014048] Yama: becoming mindful. [ 0.015036] SELinux: Initializing. [ 0.016063] *** VALIDATE selinux *** [ 0.024149] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028532] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031104] *** VALIDATE tmpfs *** [ 0.033145] *** VALIDATE proc *** [ 0.034231] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.037218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.042005] Speculative Store Bypass: Vulnerable [ 0.045069] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.047183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049022] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051009] ... generic registers: 4 [ 0.052010] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054012] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056304] rcu: Hierarchical SRCU implementation. [ 0.058395] smp: Bringing up secondary CPUs ... [ 0.059563] x86: Booting SMP configuration: [ 0.060019] .... node #0, CPUs: #1 #2 #3 [ 0.066755] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142258] node 0 deferred pages initialised in 71ms [ 0.144110] devtmpfs: initialized [ 0.145343] x86/mm: Memory block size: 128MB [ 0.147982] gcov: version magic: 0x41383552 [ 0.149167] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151275] pinctrl core: initialized pinctrl subsystem [ 0.152213] [ 0.152783] ************************************************************* [ 0.153020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154011] ** ** [ 0.155011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.157014] ** This means that this kernel is built to expose internal ** [ 0.158012] ** IOMMU data structures, which may compromise security on ** [ 0.159011] ** your system. ** [ 0.160011] ** ** [ 0.161012] ** If you see this message and you are not debugging the ** [ 0.162008] ** kernel, report this immediately to your vendor! ** [ 0.163011] ** ** [ 0.164007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165008] ************************************************************* [ 0.166577] NET: Registered protocol family 16 [ 0.167347] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170446] cpuidle: using governor menu [ 0.172808] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175539] PCI: Using configuration type 1 for base access [ 0.178121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194199] cryptd: max_cpu_qlen set to 1000 [ 0.196413] ACPI: Added _OSI(Module Device) [ 0.198107] ACPI: Added _OSI(Processor Device) [ 0.200013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207159] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213577] ACPI: Interpreter enabled [ 0.215058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217117] ACPI: Using IOAPIC for interrupt routing [ 0.218114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234863] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249608] acpiphp: Slot [2] registered [ 0.251163] acpiphp: Slot [5] registered [ 0.252092] acpiphp: Slot [6] registered [ 0.253089] acpiphp: Slot [7] registered [ 0.255095] acpiphp: Slot [8] registered [ 0.257094] acpiphp: Slot [9] registered [ 0.258113] acpiphp: Slot [10] registered [ 0.260088] acpiphp: Slot [3] registered [ 0.261089] acpiphp: Slot [4] registered [ 0.263099] acpiphp: Slot [11] registered [ 0.265094] acpiphp: Slot [12] registered [ 0.266149] acpiphp: Slot [13] registered [ 0.268115] acpiphp: Slot [14] registered [ 0.269203] acpiphp: Slot [15] registered [ 0.271075] acpiphp: Slot [16] registered [ 0.272140] acpiphp: Slot [17] registered [ 0.274100] acpiphp: Slot [18] registered [ 0.275081] acpiphp: Slot [19] registered [ 0.277072] acpiphp: Slot [20] registered [ 0.278098] acpiphp: Slot [21] registered [ 0.280093] acpiphp: Slot [22] registered [ 0.281086] acpiphp: Slot [23] registered [ 0.283179] acpiphp: Slot [24] registered [ 0.284123] acpiphp: Slot [25] registered [ 0.286092] acpiphp: Slot [26] registered [ 0.287080] acpiphp: Slot [27] registered [ 0.289079] acpiphp: Slot [28] registered [ 0.290096] acpiphp: Slot [29] registered [ 0.292143] acpiphp: Slot [30] registered [ 0.293088] acpiphp: Slot [31] registered [ 0.296054] PCI host bridge to bus 0000:00 [ 0.297023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314106] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317167] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326676] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.331077] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341013] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343594] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347834] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.353014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.365014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.371020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.377166] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.383013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423501] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.430015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.434011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.452169] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.458016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.470015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.491117] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.540605] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.550028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594166] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.604015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635030] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.636272] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637200] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.638189] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640129] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.644152] iommu: Default domain type: Passthrough [ 0.646276] SCSI subsystem initialized [ 0.647072] ACPI: bus type USB registered [ 0.648069] usbcore: registered new interface driver usbfs [ 0.649069] usbcore: registered new interface driver hub [ 0.651085] usbcore: registered new device driver usb [ 0.653148] pps_core: LinuxPPS API ver. 1 registered [ 0.654007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.657086] PTP clock support registered [ 0.659076] EDAC MC: Ver: 3.0.0 [ 0.660381] PCI: Using ACPI for IRQ routing [ 0.661683] NetLabel: Initializing [ 0.663014] NetLabel: domain hash size = 128 [ 0.664006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.665061] NetLabel: unlabeled traffic allowed by default [ 0.667101] vgaarb: loaded [ 0.668182] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.669009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.679004] clocksource: Switched to clocksource kvm-clock [ 0.759352] VFS: Disk quotas dquot_6.6.0 [ 0.760394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.762011] *** VALIDATE ramfs *** [ 0.762823] *** VALIDATE hugetlbfs *** [ 0.763766] pnp: PnP ACPI init [ 0.765502] pnp: PnP ACPI: found 6 devices [ 0.787141] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.789153] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.790456] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.791870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.793237] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.794694] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.796332] NET: Registered protocol family 2 [ 0.797948] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.801160] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.803366] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.806938] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.809182] TCP: Hash tables configured (established 65536 bind 65536) [ 0.810893] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.812763] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814436] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816281] NET: Registered protocol family 1 [ 0.817755] RPC: Registered named UNIX socket transport module. [ 0.819025] RPC: Registered udp transport module. [ 0.819980] RPC: Registered tcp transport module. [ 0.820937] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.822283] NET: Registered protocol family 44 [ 0.823321] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.824547] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825777] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827126] PCI: CLS 0 bytes, default 64 [ 0.828113] Unpacking initramfs... [ 2.128995] debug: unmapping init [mem 0xffff9bd8bcc54000-0xffff9bd8bffbffff] [ 2.132250] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.133942] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.136565] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.609527] Initialise system trusted keyrings [ 2.610824] Key type blacklist registered [ 2.612447] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.621488] zbud: loaded [ 2.623865] *** VALIDATE nfs *** [ 2.624685] *** VALIDATE nfs4 *** [ 2.625657] pstore: using deflate compression [ 2.628093] Platform Keyring initialized [ 2.718219] NET: Registered protocol family 38 [ 2.721387] Key type asymmetric registered [ 2.722792] Asymmetric key parser 'x509' registered [ 2.723965] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.726204] io scheduler mq-deadline registered [ 2.727347] io scheduler kyber registered [ 2.728387] io scheduler bfq registered [ 2.729471] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.731352] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.733059] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.734713] ACPI: Power Button [PWRF] [ 2.738819] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.744596] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.755110] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.761070] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.772126] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.798547] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.827072] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.830877] Non-volatile memory driver v1.3 [ 2.831813] Linux agpgart interface v0.103 [ 2.853119] virtio_blk virtio1: [vda] 146552 512-byte logical blocks (75.0 MB/71.6 MiB) [ 2.854874] vda: detected capacity change from 0 to 75034624 [ 2.866920] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.869332] vdb: detected capacity change from 0 to 1073741824 [ 2.883354] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.885775] vdc: detected capacity change from 0 to 2621440000 [ 2.897744] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.899475] vdd: detected capacity change from 0 to 2621440000 [ 2.912824] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.915664] vde: detected capacity change from 0 to 4294967296 [ 2.931332] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.934444] vdf: detected capacity change from 0 to 4294967296 [ 2.946423] libphy: Fixed MDIO Bus: probed [ 2.955296] usbcore: registered new interface driver usbserial_generic [ 2.957142] usbserial: USB Serial support registered for generic [ 2.958976] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.963305] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.964973] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.966792] mousedev: PS/2 mouse device common for all mice [ 2.970258] rtc_cmos 00:05: RTC can wake from S4 [ 2.972212] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.976644] rtc_cmos 00:05: registered as rtc0 [ 2.979840] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.980089] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.985139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.985954] intel_pstate: CPU model not supported [ 2.991073] hid: raw HID events driver (C) Jiri Kosina [ 2.992484] usbcore: registered new interface driver usbhid [ 2.993841] usbhid: USB HID core driver [ 2.995097] drop_monitor: Initializing network drop monitor service [ 2.996660] Initializing XFRM netlink socket [ 2.997899] NET: Registered protocol family 10 [ 3.000438] Segment Routing with IPv6 [ 3.001671] NET: Registered protocol family 17 [ 3.004316] mpls_gso: MPLS GSO support [ 3.010767] RAS: Correctable Errors collector initialized. [ 3.012991] AVX version of gcm_enc/dec engaged. [ 3.014801] AES CTR mode by8 optimization enabled [ 3.102571] sched_clock: Marking stable (3102550978, 0)->(3984724673, -882173695) [ 3.106656] registered taskstats version 1 [ 3.108391] Loading compiled-in X.509 certificates [ 3.110060] zswap: loaded using pool lzo/zbud [ 3.132501] Key type big_key registered [ 3.144931] Key type encrypted registered [ 3.146372] ima: No TPM chip found, activating TPM-bypass! [ 3.149513] ima: Allocated hash algorithm: sha1 [ 3.150936] ima: No architecture policies found [ 3.152397] evm: Initialising EVM extended attributes: [ 3.154024] evm: security.selinux [ 3.155063] evm: security.ima [ 3.156021] evm: security.capability [ 3.157208] evm: HMAC attrs: 0x1 [ 3.159351] rtc_cmos 00:05: setting system clock to 2026-08-27 01:06:55 UTC (1787792815) [ 3.164474] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.167386] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.177092] Write protecting the kernel read-only data: 28672k [ 3.179683] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.181858] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.225419] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.234517] systemd[1]: Detected virtualization kvm. [ 3.236904] systemd[1]: Detected architecture x86-64. [ 3.238543] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.263303] systemd[1]: No hostname configured. [ 3.264475] systemd[1]: Set hostname to . [ 3.265971] random: systemd: uninitialized urandom read (16 bytes read) [ 3.267980] systemd[1]: Initializing machine ID from random generator. [ 3.297980] random: ln: uninitialized urandom read (6 bytes read) [ 3.382119] random: systemd: uninitialized urandom read (16 bytes read) [ 3.384325] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.388564] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.391772] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.896589] device-mapper: uevent: version 1.0.3 [ 3.898689] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.564852] virtio_net virtio0 ens2: renamed from eth0 [ 4.601928] random: fast init done [ 4.630942] scsi host0: ata_piix [ 4.659309] scsi host1: ata_piix [ 4.660679] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.663294] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.292615] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.673350] random: crng init done [ 9.674965] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.741890] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.263153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.055903] SELinux: Disabled at runtime. [ 14.185029] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.209572] systemd[1]: Detected virtualization kvm. [ 14.211404] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.423060] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.427841] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.447440] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.463388] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.475634] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.510510] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.545287] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ 16.148870] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.900901] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.671768] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.767506] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.149646] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.306585] EDAC sbridge: Ver: 1.1.2 [ 21.746938] Key type dns_resolver registered [ 22.325804] NFS: Registering the id_resolver key type [ 22.328486] Key type id_resolver registered [ 22.330305] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 50.657476] hrtimer: interrupt took 7504812 ns [ 80.416482] libcfs: loading out-of-tree module taints kernel. [ 80.449549] Key type ._llcrypt registered [ 80.450399] Key type .llcrypt registered [ 80.529693] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 104.201289] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 106.481733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.505132] alg: No test for adler32 (adler32-zlib) [ 108.310072] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 109.445606] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 111.199380] Key type lgssc registered [ 112.878524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.573277] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.025967] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 184.445824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.478711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.648470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.676496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.731384] Lustre: lustre-MDT0000: new disk, initializing [ 185.802155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.858802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.019448] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.246765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.298939] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.326993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.328930] Lustre: Skipped 1 previous similar message [ 205.381210] Lustre: lustre-MDT0001: new disk, initializing [ 205.443172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.461379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.482592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.262500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.930713] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.567464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.789435] Lustre: lustre-OST0000: new disk, initializing [ 224.793436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.797044] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.861253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.448625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.483047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.663077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.742577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.454710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.640766] Lustre: lustre-OST0001: new disk, initializing [ 253.644118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.647966] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.713812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.550743] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.749730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.763741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.831134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.732425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.464429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.424583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.123554] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 307.428767] Lustre: DEBUG MARKER: Client: 2.17.57.48 [ 309.808537] Lustre: DEBUG MARKER: MDS: 2.17.57.48 [ 312.672816] Lustre: DEBUG MARKER: OSS: 2.17.57.48 [ 314.405758] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 21:12:05 EDT 2026 [ 333.098967] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 334.708994] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.274312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.277789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.293864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.316972] Lustre: Skipped 2 previous similar messages [ 357.554883] Lustre: server umount lustre-MDT0000 complete [ 361.671163] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793174 with bad export cookie 14187851223290693479 [ 361.678073] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 361.682343] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.513533] LustreError: 9955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.513732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.535523] LustreError: 9955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 364.558786] Lustre: Skipped 1 previous similar message [ 364.561834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.567868] Lustre: Skipped 4 previous similar messages [ 366.559617] LustreError: 9955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.561232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.571221] LustreError: 9955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 366.586990] Lustre: Skipped 1 previous similar message [ 368.285736] Lustre: server umount lustre-MDT0001 complete [ 373.062966] Lustre: server umount lustre-OST0000 complete [ 376.874184] Lustre: server umount lustre-OST0001 complete [ 384.333524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 392.868395] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 432.299609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.756664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.810448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.923417] Lustre: lustre-MDT0000: new disk, initializing [ 433.055682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.087816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 438.528891] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 449.855785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.013857] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.140526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.144266] Lustre: Skipped 1 previous similar message [ 450.314178] Lustre: lustre-MDT0001: new disk, initializing [ 450.423968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.457160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.466494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.096532] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 465.056879] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.863509] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.630915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.859155] Lustre: lustre-OST0000: new disk, initializing [ 477.862830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.867477] Lustre: 17062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.937654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.538049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.586436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.683058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.717433] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 494.940476] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.235344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 500.570411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 505.373585] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 505.581753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.440167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.455840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.465112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.481832] Lustre: Skipped 2 previous similar messages [ 513.890748] Lustre: server umount lustre-OST0000 complete [ 520.674682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.687890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.694237] Lustre: Skipped 1 previous similar message [ 520.717101] Lustre: Skipped 2 previous similar messages [ 524.614390] Lustre: server umount lustre-MDT0000 complete [ 525.800988] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.827672] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 529.708373] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793342 with bad export cookie 14187851223290695012 [ 529.719356] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.725361] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 530.170340] Lustre: server umount lustre-MDT0001 complete [ 536.555591] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:15:47 (1787793347) [ 557.693849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.127410] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.218960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.585549] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 563.682533] LustreError: 19938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.906747] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.932211] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 573.810109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.968610] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 587.489606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.053858] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.608122] Lustre: Failing over lustre-MDT0000 [ 596.999688] Lustre: server umount lustre-MDT0000 complete [ 599.718625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.731039] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.743461] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 616.415358] Lustre: 3658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793412/real 1787793412] req@ffff9bd9417a3b80 x1874636750095744/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787793428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 616.438351] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.523333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.725305] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc4e55f84625552ea [ 625.774273] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 626.200432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.213330] Lustre: Skipped 1 previous similar message [ 626.326649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 631.000515] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 631.264564] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 631.272307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 631.284855] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 641.729917] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 646.624440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.632918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.653223] Lustre: Skipped 1 previous similar message [ 646.668045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.128285] Lustre: server umount lustre-MDT0000 complete [ 655.843110] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.856408] LustreError: 19937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 657.319895] LustreError: 19918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793469 with bad export cookie 14187851223290696426 [ 657.328865] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.824669] Lustre: server umount lustre-MDT0001 complete [ 666.797905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.599541] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 683.698625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.309517] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 698.331650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 705.837890] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 708.875389] Lustre: Failing over lustre-MDT0000 [ 709.098090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.100788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.121914] Lustre: Skipped 2 previous similar messages [ 709.137966] Lustre: Skipped 3 previous similar messages [ 709.526876] Lustre: server umount lustre-MDT0000 complete [ 720.357157] LustreError: 24280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.386629] LustreError: 24280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 726.496218] Lustre: 3660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793522/real 1787793522] req@ffff9bd9041b0700 x1874636750127616/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787793538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 726.532908] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.359590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.226680] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 741.345921] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 741.352972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 741.373620] Lustre: Skipped 1 previous similar message [ 741.423187] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 741.615691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 753.298869] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 760.801200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.826031] Lustre: Skipped 1 previous similar message [ 760.830522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 763.487908] Lustre: server umount lustre-MDT0000 complete [ 767.873574] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793580 with bad export cookie 14187851223290697371 [ 767.877664] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 767.887091] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 768.356651] Lustre: server umount lustre-MDT0001 complete [ 779.664944] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:19:50 (1787793590) [ 797.834690] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 809.944447] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 855.921477] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 867.250876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.649603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 867.679396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 867.772332] Lustre: lustre-MDT0000: new disk, initializing [ 867.870221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.880457] Lustre: Skipped 3 previous similar messages [ 867.907403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 876.201895] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 888.627186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.734651] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 888.953394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 894.365493] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 905.890461] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 911.577834] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.212260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.584620] Lustre: 31629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 921.711527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 921.752481] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 921.826886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 921.841698] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 928.520062] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 940.160850] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.568359] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 968.164088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.177501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.207514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.225445] Lustre: Skipped 2 previous similar messages [ 974.066316] Lustre: server umount lustre-OST0000 complete [ 980.970239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 985.510231] Lustre: server umount lustre-MDT0000 complete [ 988.645765] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.658682] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 989.748694] LustreError: 29650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793802 with bad export cookie 14187851223290697777 [ 989.761305] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 989.770691] LustreError: 29650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 990.487133] Lustre: server umount lustre-MDT0001 complete [ 1002.238764] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1005.561251] Key type lgssc unregistered [ 1005.942914] LNet: 33609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.959099] LNetError: 33609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.972067] LNet: Removed LNI 192.168.202.119@tcp [ 1007.082943] Key type .llcrypt unregistered [ 1007.084512] Key type ._llcrypt unregistered [ 1032.778876] Key type ._llcrypt registered [ 1032.781862] Key type .llcrypt registered [ 1032.883867] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1034.080352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.252034] alg: No test for adler32 (adler32-zlib) [ 1035.309735] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1035.511110] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1037.151169] Key type lgssc registered [ 1038.250730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.481615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.525354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.099265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.934704] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1058.278917] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.294460] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1062.239668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.647206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.866976] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1077.300121] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1083.649822] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.673596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.264460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.616286] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1106.235526] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1107.699923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1121.684754] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:25:32 (1787793932) [ 1123.764277] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1125.661715] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1133.368992] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:25:43 (1787793943) [ 1143.112469] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:25:54 (1787793954) [ 1144.361200] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1154.528819] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1154.536340] Lustre: Skipped 2 previous similar messages [ 1157.102563] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:26:07 (1787793967) [ 1159.321638] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1159.324709] Lustre: Skipped 2 previous similar messages [ 1169.369493] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1169.377928] Lustre: Skipped 4 previous similar messages [ 1184.277747] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:26:35 (1787793995) [ 1187.896353] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1192.196603] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1201.254519] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:26:52 (1787794012) [ 1203.378381] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1203.385698] Lustre: Skipped 6 previous similar messages [ 1216.798251] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:27:07 (1787794027) [ 1235.942533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.943465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.962492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1235.980351] Lustre: Skipped 1 previous similar message [ 1240.738599] Lustre: server umount lustre-OST0000 complete [ 1246.176724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.178519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.203252] Lustre: Skipped 2 previous similar messages [ 1246.220385] Lustre: Skipped 1 previous similar message [ 1250.282463] Lustre: server umount lustre-MDT0000 complete [ 1251.303157] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.315607] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1253.982805] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794066 with bad export cookie 16562921306580882285 [ 1253.989178] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.001720] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1254.429765] Lustre: server umount lustre-MDT0001 complete [ 1268.670558] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1271.937721] Key type lgssc unregistered [ 1272.290666] LNet: 42881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.295746] LNetError: 42881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.320806] LNet: Removed LNI 192.168.202.119@tcp [ 1273.324409] Key type .llcrypt unregistered [ 1273.327352] Key type ._llcrypt unregistered [ 1293.039763] Key type ._llcrypt registered [ 1293.041558] Key type .llcrypt registered [ 1293.192213] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1309.970627] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1311.821034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1311.889292] alg: No test for adler32 (adler32-zlib) [ 1313.089189] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1313.367260] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1315.127173] Key type lgssc registered [ 1316.241716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1356.260560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1356.287067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.615769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1357.666373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1357.725456] Lustre: lustre-MDT0000: new disk, initializing [ 1357.814846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.835426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1362.791834] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1374.050058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.331485] Lustre: 46038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1374.449469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1374.455692] Lustre: Skipped 1 previous similar message [ 1374.530840] Lustre: lustre-MDT0001: new disk, initializing [ 1374.615222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.650195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1374.668988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1379.588151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1391.041831] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1396.796841] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.410157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.656501] Lustre: lustre-OST0000: new disk, initializing [ 1403.659383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1403.664821] Lustre: 48008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1403.735100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.638563] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1410.763232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1410.781560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1410.984473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1422.283130] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1429.073574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1429.425311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1434.612715] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1434.876880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1441.767037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.769730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.772590] LustreError: Skipped 1 previous similar message [ 1441.773231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.786144] Lustre: Skipped 1 previous similar message [ 1443.046582] Lustre: server umount lustre-OST0000 complete [ 1451.490199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1451.498392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.525863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.528658] Lustre: Skipped 1 previous similar message [ 1453.820292] Lustre: server umount lustre-MDT0000 complete [ 1457.813737] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794270 with bad export cookie 7876660037257828654 [ 1457.816456] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.824198] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1458.202531] Lustre: server umount lustre-MDT0001 complete [ 1468.885832] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:31:19 (1787794279) [ 1475.767769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.209986] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.228917] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1476.293304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.727682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1481.696240] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.806746] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.732764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.159515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.595329] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1502.922199] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.384502] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.149517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.704793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.872841] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1532.047099] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1540.087423] LustreError: 52995:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1540.941581] Lustre: Modifying parameter general.timeout=1 in log params [ 1542.010032] Lustre: Modifying parameter general.timeout=2 in log params [ 1544.856080] Lustre: Modifying parameter general.timeout=5 in log params [ 1544.859493] Lustre: Skipped 2 previous similar messages [ 1549.544124] Lustre: Modifying parameter general.timeout=10 in log params [ 1549.551387] Lustre: Skipped 4 previous similar messages [ 1557.754637] Lustre: Modifying parameter general.timeout=19 in log params [ 1557.757551] Lustre: Skipped 8 previous similar messages [ 1573.899720] Lustre: Modifying parameter general.timeout=35 in log params [ 1573.904716] Lustre: Skipped 15 previous similar messages [ 1605.954843] Lustre: Modifying parameter general.timeout=64 in log params [ 1605.957486] Lustre: Skipped 28 previous similar messages [ 1670.161809] Lustre: Modifying parameter general.timeout=127 in log params [ 1670.172776] Lustre: Skipped 62 previous similar messages [ 1682.607918] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:34:53 (1787794493) [ 1685.975220] Lustre: 59943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1687.890397] Lustre: 60054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1691.551313] Lustre: 60252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1691.570564] Lustre: 60252:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1723.469175] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1723.481555] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1723.549109] LustreError: 61828:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1729.057662] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:35:39 (1787794539) [ 1744.865482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.879456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.887241] Lustre: Skipped 3 previous similar messages [ 1744.898788] Lustre: Skipped 3 previous similar messages [ 1748.275832] Lustre: server umount lustre-MDT0000 complete [ 1749.985097] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.037493] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1752.500538] LustreError: 50450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794564 with bad export cookie 7876660037257829536 [ 1752.514421] LustreError: 50450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1752.522834] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.085958] Lustre: server umount lustre-MDT0001 complete [ 1768.361221] Lustre: server umount lustre-OST0000 complete [ 1794.336185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.700376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1795.191540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.590544] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1812.740339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.913109] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1812.953331] Lustre: 64361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1820.049631] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1827.209300] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.113441] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1840.277376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.489139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1840.748480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.761744] Lustre: Skipped 1 previous similar message [ 1849.781667] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1859.416914] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.942330] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1860.947631] Lustre: Skipped 26 previous similar messages [ 1860.984320] Lustre: 66929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1867.745607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.752484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.770694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.392312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.402172] Lustre: Skipped 3 previous similar messages [ 1873.679237] Lustre: server umount lustre-OST0000 complete [ 1878.499357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.516210] Lustre: Skipped 1 previous similar message [ 1883.619319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.634884] Lustre: Skipped 3 previous similar messages [ 1892.319157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1892.785209] Lustre: server umount lustre-MDT0000 complete [ 1893.856805] LustreError: 65769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.864304] LustreError: 65769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1896.640177] LustreError: 65102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794708 with bad export cookie 7876660037257832693 [ 1896.642240] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.649701] LustreError: 65102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1897.014551] Lustre: server umount lustre-MDT0001 complete [ 1908.166110] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1910.952789] Key type lgssc unregistered [ 1911.219460] LNet: 68140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.224721] LNetError: 68140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.239944] LNet: Removed LNI 192.168.202.119@tcp [ 1912.256444] Key type .llcrypt unregistered [ 1912.258387] Key type ._llcrypt unregistered [ 1939.434431] Key type ._llcrypt registered [ 1939.438799] Key type .llcrypt registered [ 1939.575946] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1941.069105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.109653] alg: No test for adler32 (adler32-zlib) [ 1942.363182] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1942.679305] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1944.391256] Key type lgssc registered [ 1945.639946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.768993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.795666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.698575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.823348] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1966.063864] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.099801] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1969.781648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.469526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.131593] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1987.091341] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1994.810187] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.028130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.642598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.682840] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2017.421905] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2036.614348] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2046.944305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.947985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.971294] Lustre: Skipped 2 previous similar messages [ 2046.980092] Lustre: Skipped 1 previous similar message [ 2052.066556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.072828] Lustre: Skipped 3 previous similar messages [ 2052.635073] Lustre: server umount lustre-MDT0000 complete [ 2056.569736] LustreError: 69536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794868 with bad export cookie 1889383473112003213 [ 2056.571529] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.576593] LustreError: 69536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2057.078491] Lustre: server umount lustre-MDT0001 complete [ 2071.099387] Lustre: server umount lustre-OST0000 complete [ 2099.256862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.687612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2099.746209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2099.997219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.872597] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2114.374649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.471798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2114.501824] Lustre: 74452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2114.964376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.767547] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2128.475067] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.934904] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2142.693439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2142.967031] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2143.498406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.141677] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2159.984832] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2162.322338] Lustre: 77029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2164.527668] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2169.824286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.831058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.845320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2170.341749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2175.472107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2175.475682] Lustre: Skipped 3 previous similar messages [ 2176.114540] Lustre: server umount lustre-OST0000 complete [ 2180.579456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.586144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.604460] Lustre: Skipped 2 previous similar messages [ 2180.618458] Lustre: Skipped 1 previous similar message [ 2186.847516] Lustre: server umount lustre-MDT0000 complete [ 2190.822810] LustreError: 77161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.855382] LustreError: 77161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2191.780855] LustreError: 75199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795004 with bad export cookie 1889383473112004053 [ 2191.783786] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.792119] LustreError: 75199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2192.310563] Lustre: server umount lustre-MDT0001 complete [ 2205.559276] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2208.726634] Key type lgssc unregistered [ 2209.036378] LNet: 78242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.053368] LNetError: 78242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.065960] LNet: Removed LNI 192.168.202.119@tcp [ 2210.290782] Key type .llcrypt unregistered [ 2210.292482] Key type ._llcrypt unregistered [ 2239.892553] Key type ._llcrypt registered [ 2239.901036] Key type .llcrypt registered [ 2240.080783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2241.550421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.605361] alg: No test for adler32 (adler32-zlib) [ 2242.598781] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2242.913882] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2244.583189] Key type lgssc registered [ 2246.438610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2254.237547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2254.287261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.863777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.143777] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2266.081978] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.113193] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2271.129268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.205587] LustreError: 79658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.071108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2276.965589] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2284.799581] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.284896] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.175694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.868202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.128132] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2315.403159] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2321.528131] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2322.607059] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2324.592323] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2324.598870] Lustre: Skipped 1 previous similar message [ 2327.412044] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.417181] Lustre: Skipped 2 previous similar messages [ 2331.702137] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2331.706443] Lustre: Skipped 3 previous similar messages [ 2339.835448] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.843468] Lustre: Skipped 8 previous similar messages [ 2356.050294] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.052335] Lustre: Skipped 16 previous similar messages [ 2388.132161] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2388.136708] Lustre: Skipped 30 previous similar messages [ 2432.379823] Lustre: 87771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2435.769121] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:47:26 (1787795246) [ 2438.156835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2440.320731] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:47:30 (1787795250) [ 2449.384118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.389671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.411946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.174726] Lustre: server umount lustre-OST0000 complete [ 2456.032282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2456.037467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.046946] Lustre: Skipped 1 previous similar message [ 2456.053959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2456.060268] Lustre: Skipped 1 previous similar message [ 2459.618541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.622413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.408947] Lustre: server umount lustre-MDT0000 complete [ 2464.742420] LustreError: 79658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.766084] LustreError: 79658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2465.698019] LustreError: 79639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795278 with bad export cookie 17216358495912109455 [ 2465.709630] LustreError: 79639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2465.711078] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.188988] Lustre: server umount lustre-MDT0001 complete [ 2477.806925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2480.883511] Key type lgssc unregistered [ 2481.229231] LNet: 89023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.241855] LNetError: 89023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2481.260247] LNet: Removed LNI 192.168.202.119@tcp [ 2482.176345] Key type .llcrypt unregistered [ 2482.178419] Key type ._llcrypt unregistered [ 2486.863774] Key type ._llcrypt registered [ 2486.866543] Key type .llcrypt registered [ 2486.989924] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2491.468630] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules [ 2492.093819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2492.145761] alg: No test for adler32 (adler32-zlib) [ 2493.240378] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2493.629577] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2495.287312] Key type lgssc registered [ 2496.869675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.115927] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2520.029665] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2520.438550] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2531.293260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2531.325236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.961649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.968684] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2543.073117] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.130691] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2547.021155] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:49:17 (1787795357) [ 2548.193370] LustreError: 91249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.598957] LustreError: 91281:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2551.629370] LustreError: 91281:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2551.640928] LustreError: 91281:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2552.154517] Lustre: server umount lustre-MDT0000 complete [ 2566.045407] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2569.107542] Key type lgssc unregistered [ 2569.370408] LNet: 92778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.377684] LNetError: 92778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.403820] LNet: Removed LNI 192.168.202.119@tcp [ 2570.417539] Key type .llcrypt unregistered [ 2570.423992] Key type ._llcrypt unregistered [ 2587.921732] Key type ._llcrypt registered [ 2587.923960] Key type .llcrypt registered [ 2587.995950] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2589.207162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2589.219628] alg: No test for adler32 (adler32-zlib) [ 2590.380342] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2590.653100] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2592.479290] Key type lgssc registered [ 2594.019515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2602.846678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2602.862141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.382163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.915326] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2613.742783] LustreError: 93997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.763829] LustreError: 93997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2618.436382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.269094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2624.079719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2632.033601] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2638.323672] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2645.166084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.989930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.899763] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2661.661803] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2709.410692] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:52:00 (1787795520) [ 2711.115816] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2713.647786] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:52:04 (1787795524) [ 2721.251240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.254456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.259986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.764969] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.769256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.783354] Lustre: Skipped 1 previous similar message [ 2725.524640] Lustre: server umount lustre-MDT0000 complete [ 2726.886207] LustreError: 93998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.917444] LustreError: 93998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2729.514874] LustreError: 93978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795541 with bad export cookie 6370247701990743674 [ 2729.523291] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.528661] LustreError: 93978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2729.988518] Lustre: server umount lustre-MDT0001 complete [ 2744.803871] Lustre: server umount lustre-OST0000 complete [ 2756.728679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.328915] LustreError: 98275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.467803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.682575] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2762.724289] LustreError: 98276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.817842] LustreError: 98275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.773552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.318954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.744663] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2781.828309] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2786.758324] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.629381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.801338] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2799.807971] LustreError: 98265:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2799.838152] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2799.842839] LustreError: 100358:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2799.847316] LustreError: 100358:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2799.851424] LustreError: 100358:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2799.901354] Lustre: server umount lustre-OST0000 complete [ 2799.907131] LustreError: 100358:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2808.776687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.019058] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2809.027153] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2809.055359] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2809.069187] LustreError: 100824:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2809.077247] LustreError: 100824:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2809.082783] LustreError: 100824:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2809.197899] Lustre: server umount lustre-OST0000 complete [ 2809.201413] LustreError: 100824:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2818.391586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.553562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2818.565574] Lustre: Found index 0 for lustre-OST0000, updating log [ 2818.579702] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2818.669301] Lustre: lustre-OST0000: new disk, initializing [ 2818.682680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2818.694101] Lustre: 101359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2819.088803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.434143] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2830.537970] Lustre: Failing over lustre-OST0000 [ 2830.749098] Lustre: server umount lustre-OST0000 complete [ 2836.462430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2836.468050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.504140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.516101] Lustre: Skipped 1 previous similar message [ 2840.545092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2840.548094] Lustre: Skipped 3 previous similar messages [ 2845.669205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.677089] Lustre: Skipped 1 previous similar message [ 2849.247126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2849.506377] Lustre: server umount lustre-MDT0000 complete [ 2850.785195] LustreError: 99005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2850.809070] LustreError: 99005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2853.729662] LustreError: 98257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795666 with bad export cookie 6370247701990746418 [ 2853.733303] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.748396] LustreError: 98257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2854.302123] Lustre: server umount lustre-MDT0001 complete [ 2863.604602] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:54:34 (1787795674) [ 2882.484557] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2894.593061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.891621] LustreError: 103706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.980190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.561667] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2908.866569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.489760] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2919.346943] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2926.350906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.141291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.146478] Lustre: Skipped 1 previous similar message [ 2927.159483] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2933.798464] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2934.313487] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2934.326661] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2934.335639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2939.368791] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2939.394704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2940.407511] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2940.465596] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2940.467437] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2940.479367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2943.048428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.170268] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2943.254492] Lustre: lustre-OST0001: new disk, initializing [ 2943.257896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2943.264500] Lustre: 106026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2949.687867] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2952.872760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2952.884958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2953.037778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2959.264405] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2968.788632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2973.673020] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.679860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2973.695838] Lustre: Skipped 3 previous similar messages [ 2973.707173] Lustre: Skipped 2 previous similar messages [ 2979.490401] Lustre: server umount lustre-MDT0001 complete [ 2983.928723] LustreError: 104821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.942046] LustreError: 104821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2989.925340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.106187] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2990.111197] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2990.210099] Lustre: 103694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2991.231750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2991.238499] Lustre: Skipped 1 previous similar message [ 2992.017053] LustreError: 107569:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2992.032643] LustreError: 107569:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2992.055125] LustreError: 103698:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2992.196332] LustreError: 107580:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.119@tcp: cfg command failed: rc = -17 [ 2992.203730] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2992.203730] [ 2996.206328] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2997.014112] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3007.896169] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:56:58 (1787795818) [ 3025.294237] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3041.640071] Lustre: 109640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3062.474819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3065.404421] Lustre: 110775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3144.159696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3144.166610] LustreError: Skipped 1 previous similar message [ 3144.172921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.204391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.211901] Lustre: Skipped 5 previous similar messages [ 3147.038538] Lustre: server umount lustre-MDT0000 complete [ 3149.792667] LustreError: 103702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.810289] LustreError: 103702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3151.133235] LustreError: 106039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795963 with bad export cookie 6370247701990747419 [ 3151.148354] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3151.156422] LustreError: 106039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3151.557803] Lustre: server umount lustre-MDT0001 complete [ 3166.336954] Lustre: server umount lustre-OST0000 complete [ 3181.124829] Lustre: server umount lustre-OST0001 complete [ 3187.297531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3197.796549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3206.953329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3216.983473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3238.686914] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3250.826293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.842197] Lustre: lustre-MDT0000: reset Object Index mappings [ 3251.486652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.568328] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3265.303766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.334681] Lustre: lustre-MDT0001: reset Object Index mappings [ 3270.672851] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3273.717507] Lustre: 115891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3280.531653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.945624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3291.565656] LustreError: 116243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.606149] LustreError: 116243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3291.616166] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3296.713957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3298.568328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.445684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3304.459869] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3306.565913] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3314.774042] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3318.189629] Lustre: 117766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3370.774958] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:03:01 (1787796181) [ 3399.137113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3399.146692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.152823] Lustre: Skipped 3 previous similar messages [ 3399.156688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.159322] Lustre: Skipped 3 previous similar messages [ 3403.233169] Lustre: server umount lustre-MDT0000 complete [ 3407.138982] LustreError: 114722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796219 with bad export cookie 6370247701990794487 [ 3407.143698] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.150894] LustreError: 114722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3413.899717] Lustre: server umount lustre-MDT0001 complete [ 3424.155824] Lustre: server umount lustre-OST0000 complete [ 3428.210971] LustreError: 118724:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xba:0x0].0x0 (ffff9bd808b40600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3434.545842] Lustre: server umount lustre-OST0001 complete [ 3443.102694] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3455.509298] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3495.042247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.208990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3495.229930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3495.298710] Lustre: lustre-MDT0000: new disk, initializing [ 3495.371825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3500.561023] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3507.805407] Lustre: server umount lustre-MDT0000 complete [ 3517.858795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.293452] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3518.303934] Lustre: 122036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3518.313377] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3518.464463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.474817] Lustre: Skipped 4 previous similar messages [ 3523.910288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3529.000971] Lustre: server umount lustre-MDT0000 complete [ 3536.552403] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:05:47 (1787796347) [ 3538.111721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3540.074735] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:05:51 (1787796351) [ 3560.880073] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3573.842949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.022771] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3589.062482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.269976] Lustre: 124085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3589.326587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3589.329355] Lustre: Skipped 1 previous similar message [ 3589.386727] Lustre: lustre-MDT0001: new disk, initializing [ 3589.556729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3589.580175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3594.595812] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3600.045924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3607.211800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.628284] Lustre: lustre-OST0000: new disk, initializing [ 3607.631930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3607.637693] Lustre: 125664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3609.233318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.247457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.325837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3615.270424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3628.495082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.725876] Lustre: lustre-OST0001: new disk, initializing [ 3628.729822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3628.744628] Lustre: 126535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3630.490684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3630.512403] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3630.664274] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3637.185579] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3647.083360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3650.943910] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4028.986056] Lustre: 132950:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4028.993500] Lustre: 132950:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4028.999139] Lustre: 132950:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4029.006201] Lustre: 132950:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4029.013219] Lustre: 132950:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4029.018825] Lustre: 132950:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4310.019505] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:18:40 (1787797120) [ 4326.887068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.897433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4326.908230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.908238] Lustre: Skipped 8 previous similar messages [ 4326.914782] Lustre: Skipped 8 previous similar messages [ 4331.632808] Lustre: server umount lustre-MDT0000 complete [ 4332.005746] LustreError: 127339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.028594] LustreError: 127339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4335.373177] LustreError: 124080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797147 with bad export cookie 6370247701990862800 [ 4335.377761] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.387234] LustreError: 124080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4337.120517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.127860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4337.138653] Lustre: Skipped 1 previous similar message [ 4337.151276] Lustre: Skipped 4 previous similar messages [ 4341.948179] Lustre: server umount lustre-MDT0001 complete [ 4352.367063] Lustre: server umount lustre-OST0000 complete [ 4363.022480] Lustre: server umount lustre-OST0001 complete [ 4370.546804] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4381.959875] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4424.008780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.211296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4424.232526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4424.319816] Lustre: lustre-MDT0000: new disk, initializing [ 4424.397618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.400593] Lustre: Skipped 4 previous similar messages [ 4424.421307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4429.679653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4441.464824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.713393] Lustre: 136007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4441.764959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4441.768462] Lustre: Skipped 1 previous similar message [ 4441.850532] Lustre: lustre-MDT0001: new disk, initializing [ 4441.997265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4442.009809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4447.423790] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4457.278153] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4462.996651] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4470.075028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4470.651188] Lustre: lustre-OST0000: new disk, initializing [ 4470.656941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4470.670952] Lustre: 137976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4472.107948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.124671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.376452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4480.454184] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4491.023334] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.467238] Lustre: lustre-MDD0000: changelog on [ 4504.346325] Lustre: lustre-MDD0001: changelog on [ 6221.030890] Lustre: lustre-MDD0001: changelog off [ 6222.832634] Lustre: 136012:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6223.762545] Lustre: lustre-MDD0000: changelog off [ 6233.575452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.581807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6233.615797] Lustre: Skipped 1 previous similar message [ 6233.616851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.650816] Lustre: Skipped 3 previous similar messages [ 6235.404523] Lustre: server umount lustre-OST0000 complete [ 6238.699508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.719656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.808509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.816693] Lustre: Skipped 2 previous similar messages [ 6245.019235] Lustre: server umount lustre-MDT0000 complete [ 6248.941589] LustreError: 136018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.968777] LustreError: 136018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6249.044193] LustreError: 135999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799061 with bad export cookie 6370247701991679553 [ 6249.048619] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.054590] LustreError: 135999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6249.510535] Lustre: server umount lustre-MDT0001 complete [ 6260.090923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6262.752138] Key type lgssc unregistered [ 6263.105322] LNet: 147094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6263.115952] LNetError: 147094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6263.143863] LNet: Removed LNI 192.168.202.119@tcp [ 6264.039227] Key type .llcrypt unregistered [ 6264.042237] Key type ._llcrypt unregistered [ 6282.025744] Key type ._llcrypt registered [ 6282.027927] Key type .llcrypt registered [ 6282.147943] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6297.615273] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6298.880587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6298.900112] alg: No test for adler32 (adler32-zlib) [ 6300.056110] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 6300.337402] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6302.119194] Key type lgssc registered [ 6303.413940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6339.380107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6339.417450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.928208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6341.009243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6341.155386] Lustre: lustre-MDT0000: new disk, initializing [ 6341.256872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6341.278326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6347.600279] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6359.004588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.117874] Lustre: 150293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6359.167338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6359.171260] Lustre: Skipped 1 previous similar message [ 6359.247474] Lustre: lustre-MDT0001: new disk, initializing [ 6359.320600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6359.356947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6359.368513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6364.179533] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6374.245774] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6380.700693] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6387.582256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.996793] Lustre: lustre-OST0000: new disk, initializing [ 6387.999974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6388.007665] Lustre: 152265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6388.129850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.207902] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6396.546593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6396.557773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6396.791563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6405.665558] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.503049] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_module obdecho/obdecho [ 6414.174685] LustreError: 153269:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6414.181577] LustreError: 153269:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6415.864314] LustreError: 153365:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6427.652310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.654422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6427.671570] Lustre: Skipped 2 previous similar messages [ 6427.692778] Lustre: Skipped 2 previous similar messages [ 6432.745673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.760524] Lustre: Skipped 2 previous similar messages [ 6433.438162] Lustre: server umount lustre-MDT0000 complete [ 6437.394546] LustreError: 150286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799249 with bad export cookie 5731105240143572664 [ 6437.408269] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.841671] Lustre: server umount lustre-MDT0001 complete [ 6451.130176] Lustre: server umount lustre-OST0000 complete [ 6474.228764] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6484.680632] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6523.104970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.459063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6523.500185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6523.685263] Lustre: lustre-MDT0000: new disk, initializing [ 6523.765432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.787836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6529.087649] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6541.296582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.548790] Lustre: 157081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6541.630816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6541.633775] Lustre: Skipped 1 previous similar message [ 6541.688760] Lustre: lustre-MDT0001: new disk, initializing [ 6541.757501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.789159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6541.794651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6546.905907] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6555.961866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6563.556883] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6574.848821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6575.619276] Lustre: lustre-OST0000: new disk, initializing [ 6575.622126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6575.637941] Lustre: 159052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6575.816239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.352958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6581.369546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6581.558635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6585.014888] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6596.156655] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6600.374191] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6600.564416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6604.608260] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6604.918248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6607.329266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6607.330180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.335845] LustreError: Skipped 1 previous similar message [ 6607.336811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.351163] Lustre: Skipped 1 previous similar message [ 6612.450100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.458894] Lustre: Skipped 1 previous similar message [ 6613.497106] Lustre: server umount lustre-OST0000 complete [ 6618.595430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6618.610385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.629685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6618.634868] Lustre: Skipped 1 previous similar message [ 6622.688571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.573859] Lustre: server umount lustre-MDT0000 complete [ 6630.318959] LustreError: 157074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799442 with bad export cookie 5731105240143573623 [ 6630.327186] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.332066] LustreError: 157074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6630.704226] Lustre: server umount lustre-MDT0001 complete [ 6634.713866] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:57:25 (1787799445) [ 6636.389854] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6638.651447] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:57:29 (1787799449) [ 6646.065846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.776258] LustreError: 161463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.788299] LustreError: 161463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6646.840535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.872397] LustreError: 161464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.017942] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6656.993125] LustreError: 161463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6661.179195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.090908] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6673.912256] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.543949] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.868077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.125042] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6686.144232] Lustre: Skipped 2 previous similar messages [ 6686.268021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.275835] Lustre: Skipped 1 previous similar message [ 6693.190893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6701.500903] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6707.242823] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 6707.682834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.705942] Lustre: Skipped 2 previous similar messages [ 6713.603153] Lustre: server umount lustre-MDT0000 complete [ 6717.427050] LustreError: 162705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.533458] LustreError: 161443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799529 with bad export cookie 5731105240143574512 [ 6717.551740] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.554753] LustreError: 161443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6717.925480] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.549548] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6722.580033] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6724.084907] Lustre: server umount lustre-MDT0001 complete [ 6731.817323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6746.699809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.882753] LustreError: 165306:0:(llog.c:1655:llog_backup()) MGC192.168.202.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6746.888181] Lustre: 165306:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6747.616965] LustreError: 165311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6747.646832] LustreError: 165311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6764.530045] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.559217] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6798.308554] LustreError: 165311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.339760] LustreError: 165311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6808.031362] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6808.043614] Lustre: 165306:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6849.003308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6854.141511] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6855.113223] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6862.336848] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.361446] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6864.233735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.693048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6886.894582] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4f88f9d9e9234f2d [ 6886.910693] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 6892.529114] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6894.531601] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6903.134941] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6905.157117] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6911.316574] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6913.040960] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6933.476159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.483854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6933.486824] Lustre: Skipped 2 previous similar messages [ 6933.505248] Lustre: Skipped 12 previous similar messages [ 6944.223175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6944.479923] Lustre: server umount lustre-MDT0000 complete [ 6948.105172] LustreError: 161444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799760 with bad export cookie 5731105240143580973 [ 6948.112441] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.114425] LustreError: 161444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6948.470895] Lustre: server umount lustre-MDT0001 complete [ 6962.196651] Lustre: server umount lustre-OST0000 complete [ 6985.636352] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6997.650653] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7040.572227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.878511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7040.922905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7041.016800] Lustre: lustre-MDT0000: new disk, initializing [ 7041.157102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.162335] Lustre: Skipped 1 previous similar message [ 7041.194373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7046.750031] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7058.420270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.648884] Lustre: 170429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7059.125260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7064.188254] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7074.810156] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7080.630238] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.236894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.556796] Lustre: lustre-OST0000: new disk, initializing [ 7088.562030] Lustre: Skipped 1 previous similar message [ 7088.565164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7088.569743] Lustre: Skipped 2 previous similar messages [ 7088.576495] Lustre: 172400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7095.904561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7097.930149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7097.933659] Lustre: Skipped 1 previous similar message [ 7097.939529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7098.049871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7107.592888] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7114.380768] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7114.773261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7122.165409] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7122.604367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7129.078185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.081885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7129.097104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.097113] Lustre: Skipped 6 previous similar messages [ 7129.131968] Lustre: Skipped 1 previous similar message [ 7129.145089] LustreError: Skipped 1 previous similar message [ 7139.295514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7139.594659] Lustre: server umount lustre-OST0000 complete [ 7145.953116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7151.302400] Lustre: server umount lustre-MDT0000 complete [ 7154.656959] LustreError: 171234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.682323] LustreError: 171234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 7155.148817] LustreError: 171224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799967 with bad export cookie 5731105240143597801 [ 7155.167779] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7155.174523] LustreError: 171224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7155.748972] Lustre: server umount lustre-MDT0001 complete [ 7165.919421] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:06:16 (1787799976) [ 7173.462933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.544990] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7188.605417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.787370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7201.301895] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.215365] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7214.194871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.398972] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7229.939245] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.855580] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:07:31 (1787800051) [ 7246.311436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7246.326318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.362347] Lustre: Skipped 2 previous similar messages [ 7251.705433] Lustre: server umount lustre-OST0000 complete [ 7260.641715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7260.655205] Lustre: Skipped 13 previous similar messages [ 7263.823613] Lustre: server umount lustre-MDT0000 complete [ 7267.646672] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800079 with bad export cookie 5731105240143598683 [ 7267.659650] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7267.664017] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7268.049636] Lustre: server umount lustre-MDT0001 complete [ 7279.702208] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7282.657815] Key type lgssc unregistered [ 7282.924149] LNet: 178546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7282.931913] LNetError: 178546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7282.945691] LNet: Removed LNI 192.168.202.119@tcp [ 7283.837119] Key type .llcrypt unregistered [ 7283.840101] Key type ._llcrypt unregistered [ 7303.438559] Key type ._llcrypt registered [ 7303.442584] Key type .llcrypt registered [ 7303.642023] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7305.559831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7305.775071] alg: No test for adler32 (adler32-zlib) [ 7306.992625] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 7307.538428] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7309.303334] Key type lgssc registered [ 7311.145458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.842666] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7321.875857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.504348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7342.623868] LustreError: 179792:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 7347.743906] LustreError: 179792:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7347.759970] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7347.774156] LustreError: MGC192.168.202.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7347.785109] LustreError: 179792:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7347.806716] LustreError: 179792:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7347.816089] LustreError: 179792:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7347.916349] Lustre: server umount lustre-OST0000 complete [ 7347.921202] LustreError: 179792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7357.409578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.908901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.538778] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7369.202650] LustreError: 180292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.237915] LustreError: 180292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7372.908278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.516479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7379.134993] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7386.999669] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7393.773355] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7401.842831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.159323] LustreError: 182109:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7402.166559] Lustre: 182109:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7402.353928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7410.546921] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7418.890775] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.946504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.959745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7422.968296] Lustre: Skipped 2 previous similar messages [ 7422.995717] Lustre: Skipped 2 previous similar messages [ 7427.724898] Lustre: server umount lustre-MDT0000 complete [ 7428.074741] LustreError: 180288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7428.099982] LustreError: 180288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7431.909957] LustreError: 182870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800244 with bad export cookie 3178700121126554882 [ 7431.913341] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.926282] LustreError: 182870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7432.715104] Lustre: server umount lustre-MDT0001 complete [ 7447.621382] Lustre: server umount lustre-OST0000 complete [ 7457.937410] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7469.822402] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7515.395677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.891298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7515.959924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7516.157772] Lustre: lustre-MDT0000: new disk, initializing [ 7516.333887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7516.402202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7522.372483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7533.861071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.044798] Lustre: 185634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7534.160123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7534.169478] Lustre: Skipped 1 previous similar message [ 7534.310633] Lustre: lustre-MDT0001: new disk, initializing [ 7534.488911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7534.548831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7534.569622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7540.340921] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7552.411783] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7559.620685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7568.249135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7568.573459] Lustre: lustre-OST0000: new disk, initializing [ 7568.578388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7568.584899] Lustre: 187606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7568.665093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.146395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7576.164715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7576.311685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7577.310439] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7588.267210] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7594.873617] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7595.105659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7601.241331] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7601.503361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7607.274587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.279390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7607.283911] Lustre: Skipped 1 previous similar message [ 7607.284480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.294352] Lustre: Skipped 1 previous similar message [ 7610.099363] Lustre: server umount lustre-OST0000 complete [ 7616.480966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7616.499214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.545484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7620.505435] Lustre: server umount lustre-MDT0000 complete [ 7622.636734] LustreError: 185649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.647396] LustreError: 185627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800436 with bad export cookie 3178700121126555603 [ 7624.656110] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7625.023716] Lustre: server umount lustre-MDT0001 complete [ 7634.034571] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:14:05 (1787800445) [ 7653.396260] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7657.275575] Key type lgssc unregistered [ 7657.865668] LNet: 190557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7657.889391] LNetError: 190557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.927718] LNet: Removed LNI 192.168.202.119@tcp [ 7659.132904] Key type .llcrypt unregistered [ 7659.138957] Key type ._llcrypt unregistered [ 7679.389477] Key type ._llcrypt registered [ 7679.391595] Key type .llcrypt registered [ 7679.517374] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7680.909807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7681.034332] alg: No test for adler32 (adler32-zlib) [ 7682.066662] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 7682.494887] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7684.207186] Key type lgssc registered [ 7685.640461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7695.202977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7695.240763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.721985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.382483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7707.121339] LustreError: 191777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7707.148512] LustreError: 191777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7711.083868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.411843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7717.424478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7726.246836] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7732.958310] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7742.452953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.783700] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7742.785563] LustreError: 193642:0:(llog.c:1696:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7742.797536] Lustre: 193642:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7742.956811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.413225] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7758.667698] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7761.384749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.396687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.403454] Lustre: Skipped 1 previous similar message [ 7761.411946] Lustre: Skipped 2 previous similar messages [ 7766.497812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.512310] Lustre: Skipped 2 previous similar messages [ 7767.490961] Lustre: server umount lustre-MDT0000 complete [ 7771.074288] LustreError: 191758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800583 with bad export cookie 16539292958063438871 [ 7771.085707] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.092989] LustreError: 191758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7771.616486] LustreError: 191772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.621301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.640789] LustreError: 192507:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003e66404d x1874644690825344/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7771.669273] Lustre: Skipped 1 previous similar message [ 7771.712208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7771.862115] Lustre: server umount lustre-MDT0001 complete [ 7778.184363] Lustre: server umount lustre-OST0000 complete [ 7788.398544] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7798.688950] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7836.558242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.906810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7836.963841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7837.079615] Lustre: lustre-MDT0000: new disk, initializing [ 7837.175969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7837.202924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7843.663049] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7854.641236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.909284] Lustre: 197168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7854.980266] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7854.984512] Lustre: Skipped 1 previous similar message [ 7854.987871] LustreError: 197950:0:(llog.c:1696:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7854.995269] LustreError: 197950:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7855.001156] Lustre: 197950:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7855.019384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7855.027802] Lustre: Skipped 1 previous similar message [ 7855.132911] Lustre: lustre-MDT0001: new disk, initializing [ 7855.258660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7855.350246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7855.375159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7860.113942] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7871.178937] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7877.251662] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7885.062351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7885.297802] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7885.300753] Lustre: Skipped 1 previous similar message [ 7885.303805] LustreError: 199111:0:(llog.c:1696:llog_backup()) MGC192.168.202.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7885.310427] LustreError: 199111:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7885.315686] Lustre: 199111:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7885.323559] Lustre: 199111:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7885.346895] Lustre: lustre-OST0000: new disk, initializing [ 7885.352508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7885.358675] Lustre: 199135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7885.511052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7893.013538] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7893.591426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7893.620319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7893.782133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7904.366896] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7909.780716] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7910.124647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7915.471389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7915.816537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7919.595830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7919.599258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.616440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7919.621971] Lustre: Skipped 1 previous similar message [ 7919.650177] Lustre: Skipped 1 previous similar message [ 7924.723412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7924.737792] Lustre: Skipped 1 previous similar message [ 7932.385166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.641050] Lustre: server umount lustre-OST0000 complete [ 7937.504412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7937.510453] LustreError: Skipped 1 previous similar message [ 7937.514379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7937.541371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.545844] Lustre: Skipped 2 previous similar messages [ 7943.703320] Lustre: server umount lustre-MDT0000 complete [ 7945.190244] LustreError: 197179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.207316] LustreError: 197179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7947.509657] LustreError: 199134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800759 with bad export cookie 16539292958063439585 [ 7947.512482] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.522095] LustreError: 199134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.983786] Lustre: server umount lustre-MDT0001 complete [ 7957.993342] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:19:28 (1787800768) [ 7977.826402] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7988.487361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.052914] LustreError: 202265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.107761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.337204] LustreError: 202266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.732683] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8004.050763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.374212] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8012.720755] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8021.177661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8021.615953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.624559] Lustre: Skipped 1 previous similar message [ 8029.361872] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8041.772500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.013581] Lustre: lustre-OST0001: new disk, initializing [ 8042.026395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8042.053521] Lustre: 204573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8050.383093] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8052.371718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8052.389863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8052.536422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8063.012778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8070.774662] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8072.807872] Lustre: DEBUG MARKER: ADD OST3 [ 8079.220883] loop: module loaded [ 8079.417654] loop0: detected capacity change from 0 to 204800000 [ 8079.810153] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8079.837896] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8079.857269] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8089.746449] Lustre: DEBUG MARKER: STOP OST3 [ 8090.871893] loop0: detected capacity change from 0 to 204800000 [ 8100.173931] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.412326] Lustre: lustre-OST0003: new disk, initializing [ 8100.418109] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8100.427272] Lustre: 206657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8100.686151] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8100.700608] Lustre: Skipped 1 previous similar message [ 8102.328913] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8102.611310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8102.633889] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8102.687486] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8103.370987] Lustre: Failing over lustre-OST0003 [ 8103.623689] Lustre: server umount lustre-OST0003 complete [ 8104.353519] LustreError: 206762:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8107.037766] LustreError: 203752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.055957] LustreError: 203752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8108.018594] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8108.032594] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.051759] Lustre: Skipped 2 previous similar messages [ 8110.614672] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8122.806932] Lustre: DEBUG MARKER: START OST3 again [ 8123.370903] LustreError: 203749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.409691] LustreError: 203749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8132.007272] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8132.426024] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8132.604829] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8133.805913] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8133.807265] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8140.070101] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8162.332145] Lustre: Failing over lustre-OST0003 [ 8162.421239] Lustre: server umount lustre-OST0003 complete [ 8163.318657] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.336866] LustreError: 203751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.336877] LustreError: 203751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8163.363081] Lustre: Skipped 1 previous similar message [ 8183.776582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.786100] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.789344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.789352] Lustre: Skipped 2 previous similar messages [ 8183.814717] Lustre: Skipped 3 previous similar messages [ 8197.087196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8197.290977] Lustre: server umount lustre-MDT0000 complete [ 8201.635175] LustreError: 202246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801013 with bad export cookie 16539292958063440467 [ 8201.648318] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8201.659379] LustreError: 202246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8208.201478] Lustre: server umount lustre-MDT0001 complete [ 8212.712659] Lustre: server umount lustre-OST0000 complete [ 8216.948563] Lustre: server umount lustre-OST0001 complete [ 8223.288325] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8232.429783] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8273.059308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.263295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8273.290840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8273.381621] Lustre: lustre-MDT0000: new disk, initializing [ 8273.561164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.566927] Lustre: Skipped 1 previous similar message [ 8273.580741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8279.096288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8290.706182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.898487] Lustre: 212186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8291.023277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8291.027509] Lustre: Skipped 1 previous similar message [ 8291.101745] Lustre: lustre-MDT0001: new disk, initializing [ 8291.240733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8291.251271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8296.451390] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8307.085759] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8313.006988] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8321.015859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.292316] Lustre: 214157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8322.396780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8322.648417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8329.163528] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8341.561031] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8346.653662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8346.920665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8353.075462] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8353.391270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8358.372854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.374679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.378310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8358.378321] Lustre: Skipped 15 previous similar messages [ 8358.401531] Lustre: Skipped 3 previous similar messages [ 8358.441791] LustreError: Skipped 1 previous similar message [ 8361.877657] Lustre: server umount lustre-OST0000 complete [ 8368.097833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.383134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8380.970110] Lustre: server umount lustre-MDT0000 complete [ 8382.947212] LustreError: 213055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8382.977799] LustreError: 213055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 8384.840915] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801197 with bad export cookie 16539292958063442763 [ 8384.853378] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8384.862012] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8385.223960] Lustre: server umount lustre-MDT0001 complete [ 8388.761076] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:26:39 (1787801199) [ 8405.490313] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8415.841794] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8458.954310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.176450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8459.208592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8459.213457] Lustre: Skipped 1 previous similar message [ 8459.285851] Lustre: lustre-MDT0000: new disk, initializing [ 8459.289695] Lustre: Skipped 1 previous similar message [ 8459.373526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8459.379311] Lustre: Skipped 1 previous similar message [ 8464.862379] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8476.654273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.772431] Lustre: 218769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8477.019465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8481.793654] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8493.072126] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8500.423657] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8508.671917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.027941] Lustre: 220741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8510.215697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8510.370251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8516.078566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8527.290134] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8532.759293] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8532.971444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8538.023611] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8538.273201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8541.152709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.160630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.170136] LustreError: Skipped 1 previous similar message [ 8541.188075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.203229] Lustre: Skipped 3 previous similar messages [ 8541.205812] Lustre: Skipped 7 previous similar messages [ 8546.700420] Lustre: server umount lustre-OST0000 complete [ 8565.215279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8565.480579] Lustre: server umount lustre-MDT0000 complete [ 8566.766917] LustreError: 218783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.794686] LustreError: 218783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8569.678764] LustreError: 218764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801382 with bad export cookie 16539292958063443645 [ 8569.684088] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.688143] LustreError: 218764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8570.245913] Lustre: server umount lustre-MDT0001 complete [ 8578.789925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.347402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.351795] Lustre: Skipped 5 previous similar messages [ 8584.287189] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8592.682207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8598.993544] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8607.012154] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8612.463027] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8619.971106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.746569] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8633.926834] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8648.161779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.935148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8660.236517] Lustre: server umount lustre-OST0000 complete [ 8673.762579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.778048] Lustre: Skipped 19 previous similar messages [ 8678.367217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8678.560712] Lustre: server umount lustre-MDT0000 complete [ 8683.227046] LustreError: 222942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801495 with bad export cookie 16539292958063444527 [ 8683.242672] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8683.731537] Lustre: server umount lustre-MDT0001 complete [ 8773.399953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8776.675557] Key type lgssc unregistered [ 8776.941804] LNet: 226628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8776.949684] LNetError: 226628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8778.025615] LNet: Removed LNI 192.168.202.119@tcp [ 8779.120284] Key type .llcrypt unregistered [ 8779.123867] Key type ._llcrypt unregistered [ 8803.527631] Key type ._llcrypt registered [ 8803.536714] Key type .llcrypt registered [ 8803.664282] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8819.359287] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8820.977733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8821.035225] alg: No test for adler32 (adler32-zlib) [ 8822.149491] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8822.401402] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8824.119190] Key type lgssc registered [ 8825.403956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8861.307081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8861.324569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.630547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8862.718567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8862.819883] Lustre: lustre-MDT0000: new disk, initializing [ 8862.974604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.996903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8867.676614] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8881.207553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.452667] Lustre: 229982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8881.518420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8881.530079] Lustre: Skipped 1 previous similar message [ 8881.681117] Lustre: lustre-MDT0001: new disk, initializing [ 8881.879879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8881.947391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8881.977234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8887.043394] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8898.630125] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8905.552130] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.028817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.271592] Lustre: lustre-OST0000: new disk, initializing [ 8912.276533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8912.284198] Lustre: 231954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8912.351263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.108185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8918.129665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8918.280533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8918.471401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8928.538951] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8934.995248] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8935.353284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8940.695478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8940.934621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8944.097917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.097970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.107874] LustreError: Skipped 1 previous similar message [ 8944.131900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.136040] Lustre: Skipped 1 previous similar message [ 8944.153398] Lustre: Skipped 1 previous similar message [ 8949.223205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8949.234435] Lustre: Skipped 1 previous similar message [ 8949.674697] Lustre: server umount lustre-OST0000 complete [ 8953.826451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8953.832709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.842900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.462146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.481957] Lustre: Skipped 3 previous similar messages [ 8964.578389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8964.595171] Lustre: Skipped 1 previous similar message [ 8968.161623] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8968.694914] Lustre: server umount lustre-MDT0000 complete [ 8972.829454] LustreError: 229975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801785 with bad export cookie 14703713491466571554 [ 8972.852362] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.141846] Lustre: server umount lustre-MDT0001 complete [ 8992.145464] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9004.468208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.729108] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9004.736804] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9004.767411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9009.613903] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9010.147550] LustreError: 234890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.265497] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.347486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.654741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.445530] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9025.302544] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9031.917056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.312379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9038.036832] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9046.658592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9046.917060] Lustre: lustre-OST0001: new disk, initializing [ 9046.922729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9046.934149] Lustre: 237196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9053.157589] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9054.281913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9054.289869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9054.404113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9064.653857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9076.759903] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9090.438248] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:38:20 (1787801900) [ 9094.627039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.636189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.642219] Lustre: Skipped 1 previous similar message [ 9094.644789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9099.465889] Lustre: server umount lustre-MDT0000 complete [ 9100.799183] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.826868] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9103.912150] LustreError: 234870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801916 with bad export cookie 14703713491466572436 [ 9103.916415] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9103.930256] LustreError: 234870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9104.242505] Lustre: server umount lustre-MDT0001 complete [ 9117.942493] Lustre: server umount lustre-OST0000 complete [ 9121.761150] Lustre: 228205:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787801918/real 1787801918] req@ffff9bd8024af100 x1874645886288768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787801934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9121.815584] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.831603] Lustre: Skipped 3 previous similar messages [ 9123.081472] Lustre: server umount lustre-OST0001 complete [ 9130.321624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9141.016535] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9177.411516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.670380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9177.694631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9177.766380] Lustre: lustre-MDT0000: new disk, initializing [ 9177.881041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.883552] Lustre: Skipped 1 previous similar message [ 9177.918351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9182.894767] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9194.073797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.198478] Lustre: 241144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9194.258726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9194.262212] Lustre: Skipped 1 previous similar message [ 9194.319237] Lustre: lustre-MDT0001: new disk, initializing [ 9194.422487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9194.436935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9198.742187] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9208.000939] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.026232] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9220.243663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9220.544109] Lustre: lustre-OST0000: new disk, initializing [ 9220.548559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9220.565968] Lustre: 243113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9220.660119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.665595] Lustre: Skipped 1 previous similar message [ 9222.318528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9222.329276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9222.436149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9226.422154] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9235.606109] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9240.501060] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9240.803193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9245.747508] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9246.189231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9253.366471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9253.367093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.384805] LustreError: Skipped 1 previous similar message [ 9253.396615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9253.409949] Lustre: Skipped 1 previous similar message [ 9253.429372] Lustre: Skipped 5 previous similar messages [ 9255.290900] Lustre: server umount lustre-OST0000 complete [ 9261.028096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9263.584997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9263.593295] Lustre: Skipped 1 previous similar message [ 9273.312038] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9273.522296] Lustre: server umount lustre-MDT0000 complete [ 9276.782691] LustreError: 241135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802089 with bad export cookie 14703713491466574571 [ 9276.788163] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.791789] LustreError: 241135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9277.082798] Lustre: server umount lustre-MDT0001 complete [ 9286.359395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.007811] LustreError: 245331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9287.115116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.955836] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9297.410302] LustreError: 245332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.437695] LustreError: 245332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9303.967168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.900991] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9318.186637] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9324.567882] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9331.808135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.445988] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9346.357143] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9492.963633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9492.981883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.017081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9493.032528] Lustre: Skipped 7 previous similar messages [ 9507.298395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9507.441886] Lustre: server umount lustre-OST0000 complete [ 9517.076839] Lustre: server umount lustre-MDT0000 complete [ 9520.976611] LustreError: 245312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802333 with bad export cookie 14703713491466575453 [ 9520.992514] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9521.331364] Lustre: server umount lustre-MDT0001 complete [ 9615.738818] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9620.116769] Key type lgssc unregistered [ 9620.580563] LNet: 248943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9620.590510] LNetError: 248943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9620.622708] LNet: Removed LNI 192.168.202.119@tcp [ 9621.827179] Key type .llcrypt unregistered [ 9621.829795] Key type ._llcrypt unregistered [ 9651.756893] Key type ._llcrypt registered [ 9651.761143] Key type .llcrypt registered [ 9651.915925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9671.635291] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9673.679375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9673.741718] alg: No test for adler32 (adler32-zlib) [ 9674.902876] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 9675.239236] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9676.996071] Key type lgssc registered [ 9678.806695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9733.213665] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9744.069080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9744.111846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.390241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9745.440171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9745.525637] Lustre: lustre-MDT0000: new disk, initializing [ 9745.616338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.640650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9751.440152] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9761.976054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.202948] Lustre: 253013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9762.305027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9762.311072] Lustre: Skipped 1 previous similar message [ 9762.377232] Lustre: lustre-MDT0001: new disk, initializing [ 9762.522919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9762.566735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9762.577827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9767.563350] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9773.105912] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9782.812272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.231335] Lustre: lustre-OST0000: new disk, initializing [ 9783.238095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9783.246257] Lustre: 254648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9783.393543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.560238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9788.588603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9788.748971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9792.414402] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9803.661928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9803.764228] Lustre: lustre-OST0001: new disk, initializing [ 9803.767337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9803.771710] Lustre: 255520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9803.837939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9809.928595] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9813.167673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9813.180835] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9813.244305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9820.872978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9826.414327] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9839.104769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.114211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9839.117958] Lustre: Skipped 2 previous similar messages [ 9839.140281] Lustre: Skipped 3 previous similar messages [ 9843.769076] Lustre: server umount lustre-MDT0000 complete [ 9847.490992] LustreError: 254647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802659 with bad export cookie 3582196158165666883 [ 9847.500453] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.505030] LustreError: 254647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9847.819571] Lustre: server umount lustre-MDT0001 complete [ 9862.018764] Lustre: server umount lustre-OST0000 complete [ 9864.672586] Lustre: 250523:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787802661/real 1787802661] req@ffff9bd9229e0380 x1874646780683520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787802677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9864.716044] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.729273] Lustre: Skipped 1 previous similar message [ 9867.651833] Lustre: server umount lustre-OST0001 complete [ 9883.138544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9893.010857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9901.918735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9912.892034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9936.597336] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9949.885108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.901679] Lustre: lustre-MDT0000: reset Object Index mappings [ 9950.707227] LustreError: 260533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9950.728881] LustreError: 260533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9950.818958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.807699] LustreError: 260534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.902311] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9960.934705] LustreError: 260533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.051050] LustreError: 260534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.310406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9966.360435] Lustre: lustre-MDT0001: reset Object Index mappings [ 9967.227597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9972.068284] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9976.168775] Lustre: 261676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9983.589443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.807563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.861555] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9991.976109] LustreError: 262030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9996.118141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10001.084576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10006.570520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10008.402740] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10017.020796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10021.023827] Lustre: 263547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10026.980318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.993605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10026.998174] Lustre: Skipped 1 previous similar message [10032.114605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10032.126039] Lustre: Skipped 6 previous similar messages [10037.221590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.235588] Lustre: Skipped 3 previous similar messages [10041.311139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10041.595439] Lustre: server umount lustre-MDT0000 complete [10042.340906] LustreError: 261266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.368455] LustreError: 261266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10045.603847] LustreError: 260515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802857 with bad export cookie 3582196158165673673 [10045.606042] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.615929] LustreError: 260515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10045.934987] Lustre: server umount lustre-MDT0001 complete [10060.528478] Lustre: server umount lustre-OST0000 complete [10062.821330] Lustre: 250521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787802859/real 1787802859] req@ffff9bd93d867b80 x1874646780736768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787802875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10062.840816] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.855080] Lustre: Skipped 2 previous similar messages [10063.898903] Lustre: server umount lustre-OST0001 complete [10083.308587] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:54:54 (1787802894) [10103.660812] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10114.755889] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10157.414129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.627478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10157.651178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10157.735530] Lustre: lustre-MDT0000: new disk, initializing [10157.851363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.855378] Lustre: Skipped 1 previous similar message [10157.876358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10163.434093] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10174.201772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.362211] Lustre: 267801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10174.424673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10174.428708] Lustre: Skipped 1 previous similar message [10174.500514] Lustre: lustre-MDT0001: new disk, initializing [10174.643984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10174.660587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10179.972992] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10189.919892] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10196.489584] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10203.861684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.281148] Lustre: lustre-OST0000: new disk, initializing [10204.283687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10204.298993] Lustre: 269774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10206.019865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10206.033069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10206.174954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10211.697702] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10223.707537] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10229.997338] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10230.196618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10235.216744] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10235.444253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10242.023803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10242.029985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.037823] Lustre: Skipped 1 previous similar message [10242.038769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10242.047189] Lustre: Skipped 1 previous similar message [10243.509840] Lustre: server umount lustre-OST0000 complete [10251.745509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10251.764400] LustreError: Skipped 1 previous similar message [10251.768305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.775213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10253.992334] Lustre: server umount lustre-MDT0000 complete [10257.385985] LustreError: 267811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.412939] LustreError: 267811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10259.414173] LustreError: 267794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803071 with bad export cookie 3582196158165675766 [10259.429084] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10259.463178] LustreError: 267794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10260.470546] Lustre: server umount lustre-MDT0001 complete [10285.130173] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10296.909251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10297.414742] LustreError: 272709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.520140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10297.542608] Lustre: Skipped 2 previous similar messages [10303.548936] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10313.358719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.984731] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10322.000517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10329.201622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.385152] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10344.508902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.671927] Lustre: lustre-OST0001: new disk, initializing [10344.674067] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10344.677597] Lustre: 275019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10346.228996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10346.243772] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10346.390303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10350.531670] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10359.641875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10363.601480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10369.069948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10369.117340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10375.142138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10375.145402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10375.153752] Lustre: Skipped 1 previous similar message [10375.159883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.164752] Lustre: Skipped 2 previous similar messages [10375.416217] Lustre: server umount lustre-MDT0000 complete [10377.203964] LustreError: 273439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10377.246625] LustreError: 273439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10379.295255] LustreError: 275018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803191 with bad export cookie 3582196158165676648 [10379.300960] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.305821] LustreError: 275018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10379.740396] Lustre: server umount lustre-MDT0001 complete [10393.605618] Lustre: server umount lustre-OST0000 complete [10408.181724] Lustre: server umount lustre-OST0001 complete [10429.269656] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10444.399460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.314751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10445.329524] Lustre: Skipped 3 previous similar messages [10450.672671] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10459.173439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.530297] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10467.866919] Lustre: 279227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10474.715525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10481.833319] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10490.096085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.454613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10491.460481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10495.822500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10505.176832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10509.441652] Lustre: 281109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10515.725101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10515.776759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10528.056708] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:02:18 (1787803338) [10546.659930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10546.669208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.686598] Lustre: Skipped 3 previous similar messages [10546.700447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10549.785502] Lustre: server umount lustre-MDT0000 complete [10552.808673] LustreError: 278073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.838655] LustreError: 278073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10554.199600] LustreError: 278053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803366 with bad export cookie 3582196158165682948 [10554.211963] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10554.217289] LustreError: 278053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10560.732796] Lustre: server umount lustre-MDT0001 complete [10571.515557] Lustre: server umount lustre-OST0000 complete [10576.402730] Lustre: server umount lustre-OST0001 complete [10584.907792] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10597.299157] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10648.897742] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10659.588541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.888188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10659.949286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10660.019569] Lustre: lustre-MDT0000: new disk, initializing [10660.215882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10666.016830] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10681.980939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.192407] Lustre: 285349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10682.754923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10689.079433] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10694.929752] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10704.830953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10705.229852] Lustre: 286981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10705.424839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10705.435875] Lustre: Skipped 5 previous similar messages [10707.028243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10712.602291] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10724.069302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10724.265152] Lustre: 287854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10725.686989] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10725.700894] Lustre: Skipped 1 previous similar message [10725.824399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10731.945710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10741.525587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10761.257806] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:06:11 (1787803571) [10764.771335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10764.783332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.802351] Lustre: Skipped 5 previous similar messages [10764.813069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.822202] Lustre: Skipped 8 previous similar messages [10766.202797] Lustre: server umount lustre-MDT0000 complete [10770.779954] LustreError: 286980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803583 with bad export cookie 3582196158165690816 [10770.781935] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10770.789963] LustreError: 286980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10771.166098] Lustre: server umount lustre-MDT0001 complete [10785.646899] Lustre: server umount lustre-OST0000 complete [10788.513794] Lustre: 250521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787803584/real 1787803584] req@ffff9bd80fa27800 x1874646780995840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787803600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10790.097759] Lustre: server umount lustre-OST0001 complete [10800.331754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.984676] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10810.854407] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.923762] LustreError: 290103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10817.740987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.460724] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10831.952741] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.126833] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10842.893156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.319985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10859.949117] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10879.028379] loop1: detected capacity change from 0 to 204800000 [10879.241861] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10879.253454] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10879.280938] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.100108] loop1: detected capacity change from 0 to 204800000 [10889.780092] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.869956] LustreError: 290090:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14690: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10889.887069] LustreError: 290090:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14690 [10889.893844] LustreError: 290090:0:(mgs_handler.c:517:mgs_target_reg()) fs14690-MDT0001: Error checking target index: rc = -13 [10889.901955] LustreError: fs14690-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10889.910575] LustreError: 293821:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10889.916595] LustreError: 293821:0:(tgt_mount.c:2129:server_put_super()) no obd fs14690-MDT0001 [10889.921231] LustreError: 293821:0:(tgt_mount.c:116:server_deregister_mount()) fs14690-MDT0001 not registered [10889.942480] Lustre: server umount fs14690-MDT0001 complete [10889.946814] LustreError: 293821:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10898.406136] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.486260] Lustre: Setting parameter fs14690-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14690-MDT0001 [10898.497770] Lustre: Skipped 1 previous similar message [10898.567088] Lustre: srv-fs14690-MDT0001: No data found on store. Initialize space. [10898.576129] Lustre: Skipped 4 previous similar messages [10898.781126] Lustre: fs14690-MDT0001: new disk, initializing [10898.784285] Lustre: Skipped 3 previous similar messages [10899.003792] Lustre: srv-fs14690-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10899.013651] Lustre: 294288:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14690-MDT0001-mdtlov: can't create new llog: rc = -115 [10899.028283] LustreError: 294288:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14690-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10903.698366] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10910.777460] Lustre: server umount fs14690-MDT0001 complete [10926.575460] loop1: detected capacity change from 0 to 204800000 [10928.120847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10932.301677] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10932.979642] Lustre: 295856:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14690-MDT0001-mdtlov: can't create new llog: rc = -115 [10932.989691] LustreError: 295856:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14690-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10938.369593] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10945.382578] Lustre: server umount fs14690-MDT0001 complete [10958.359394] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [10958.821039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.847480] Lustre: Skipped 6 previous similar messages [10963.036859] Lustre: server umount lustre-MDT0000 complete [10972.248925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.412191] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10977.696541] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10978.308552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10985.366412] loop1: detected capacity change from 0 to 204800000 [10990.405373] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.661541] Lustre: 298303:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14690-MDT0001-mdtlov: can't create new llog: rc = -115 [10990.681463] LustreError: 298303:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14690-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10994.937675] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10999.917751] Lustre: server umount fs14690-MDT0001 complete [11025.377100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11025.387257] LustreError: Skipped 1 previous similar message [11030.839832] Lustre: server umount lustre-OST0000 complete [11040.929342] Lustre: server umount lustre-MDT0000 complete [11045.001100] LustreError: 290082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803857 with bad export cookie 3582196158165693497 [11045.003580] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.011973] LustreError: 290082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11045.500462] Lustre: server umount lustre-MDT0001 complete [11063.816389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11066.548492] Key type lgssc unregistered [11066.870666] LNet: 300672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11066.878466] LNetError: 300672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11067.942132] LNet: Removed LNI 192.168.202.119@tcp [11069.116329] Key type .llcrypt unregistered [11069.119139] Key type ._llcrypt unregistered [11089.816908] Key type ._llcrypt registered [11089.823576] Key type .llcrypt registered [11090.021722] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11092.275151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11092.341292] alg: No test for adler32 (adler32-zlib) [11093.910425] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11094.510374] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11096.455194] Key type lgssc registered [11099.048308] Lustre: Echo OBD driver; http://www.lustre.org/ [11108.846516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11108.857264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11110.134378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11115.217290] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11120.620768] LustreError: 301945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.645228] LustreError: 301945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11125.186893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.831901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11131.202988] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11139.098602] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11144.459446] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11153.106299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11153.835170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11154.870460] LustreError: 303774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.046600] LustreError: 303775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.091084] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11165.165776] LustreError: 303825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.545347] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11170.286318] LustreError: 303774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.524641] LustreError: 303776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.544609] LustreError: 303776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11190.474627] loop1: detected capacity change from 0 to 204800000 [11190.658048] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11190.690638] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11190.712511] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11195.549498] loop1: detected capacity change from 0 to 204800000 [11201.030613] LustreError: 303775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.073269] LustreError: 303775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11201.625190] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11201.776653] Lustre: fs7835-OST0000: new disk, initializing [11201.778836] Lustre: srv-fs7835-OST0000: No data found on store. Initialize space. [11201.782861] Lustre: 305683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11201.845560] Lustre: fs7835-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11207.267710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11218.616771] Lustre: server umount fs7835-OST0000 complete [11236.835159] LustreError: 303775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.847873] LustreError: 303775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11236.864674] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11239.311661] loop1: detected capacity change from 0 to 204800000 [11246.859921] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.335081] Lustre: fs7835-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11252.215513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11254.381270] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11264.471971] loop2: detected capacity change from 0 to 204800000 [11264.725074] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11264.734517] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11264.754692] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11272.181177] loop2: detected capacity change from 0 to 204800000 [11278.996994] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.131593] LustreError: fs7835-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11279.146247] LustreError: fs7835-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11279.159290] LustreError: 308725:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11279.161949] LustreError: 308725:0:(tgt_mount.c:2129:server_put_super()) no obd fs7835-OST0001 [11279.179367] LustreError: 308725:0:(tgt_mount.c:116:server_deregister_mount()) fs7835-OST0001 not registered [11279.201930] Lustre: server umount fs7835-OST0001 complete [11279.204157] LustreError: 308725:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11291.707419] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.823406] Lustre: fs7835-OST0001: new disk, initializing [11291.827695] Lustre: srv-fs7835-OST0001: No data found on store. Initialize space. [11291.832975] Lustre: 309277:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11291.918049] Lustre: fs7835-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11299.398744] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11303.434164] LustreError: 303776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.459280] LustreError: 303776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [11311.288653] Lustre: server umount fs7835-OST0000 complete [11321.476462] Lustre: server umount fs7835-OST0001 complete [11336.848808] loop1: detected capacity change from 0 to 204800000 [11342.731028] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11343.189035] Lustre: fs7835-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11349.969191] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11357.700793] loop2: detected capacity change from 0 to 204800000 [11364.039564] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.550215] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11380.356959] Lustre: server umount fs7835-OST0000 complete [11388.391070] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11391.077433] Lustre: server umount fs7835-OST0001 complete [11403.752366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.762985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11403.766218] Lustre: Skipped 1 previous similar message [11403.770702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11403.781678] Lustre: Skipped 1 previous similar message [11408.866212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11410.006768] Lustre: server umount lustre-OST0000 complete [11417.569723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11417.580868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.601411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.619411] Lustre: Skipped 1 previous similar message [11419.109967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.048655] Lustre: server umount lustre-MDT0000 complete [11425.159091] LustreError: 301927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804237 with bad export cookie 681041831574880763 [11425.167416] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11425.179073] LustreError: 301927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11425.730694] Lustre: server umount lustre-MDT0001 complete [11438.342424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11441.849073] Key type lgssc unregistered [11442.197663] LNet: 315206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11442.204982] LNetError: 315206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11442.224348] LNet: Removed LNI 192.168.202.119@tcp [11443.492162] Key type .llcrypt unregistered [11443.497421] Key type ._llcrypt unregistered [11478.951262] Key type ._llcrypt registered [11478.954575] Key type .llcrypt registered [11479.081796] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11495.828308] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11496.905653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11497.053546] alg: No test for adler32 (adler32-zlib) [11498.293155] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11498.514428] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11500.159181] Key type lgssc registered [11501.326856] Lustre: Echo OBD driver; http://www.lustre.org/ [11542.458721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11542.478669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.743518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11543.790975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11543.998942] Lustre: lustre-MDT0000: new disk, initializing [11544.133218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11544.163568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11548.544107] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11558.647535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.806941] Lustre: 319065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11558.869733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11558.877091] Lustre: Skipped 1 previous similar message [11558.963839] Lustre: lustre-MDT0001: new disk, initializing [11559.095859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11559.144371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11559.159655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11563.919923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11574.372222] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11581.006048] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11587.614159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11587.891584] Lustre: lustre-OST0000: new disk, initializing [11587.895723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11587.904128] Lustre: 321034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11587.996742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.771714] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11594.827202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11594.841251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11594.935054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11603.767745] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11605.814543] Lustre: Mounted lustre-client - version 2.17.57_48_g018c4cb [11606.722207] Lustre: Unmounted lustre-client [11615.712633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11615.719728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.720551] LustreError: Skipped 1 previous similar message [11615.721477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11615.754560] Lustre: Skipped 1 previous similar message [11616.225840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11616.240739] Lustre: Skipped 2 previous similar messages [11621.349804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11621.611553] Lustre: server umount lustre-OST0000 complete [11625.966550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11625.975089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.989677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.992237] Lustre: Skipped 1 previous similar message [11631.740067] Lustre: server umount lustre-MDT0000 complete [11634.830667] LustreError: 319057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804447 with bad export cookie 15406897625799672996 [11634.832236] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11634.838375] LustreError: 319057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11635.132086] Lustre: server umount lustre-MDT0001 complete [11644.883577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11646.981537] Key type lgssc unregistered [11667.695158] LNet: 323199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11667.703399] LNetError: 323199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11736.357773] LNet: Removed LNI 192.168.202.119@tcp [11737.173965] Key type .llcrypt unregistered [11737.178116] Key type ._llcrypt unregistered [11754.085758] Key type ._llcrypt registered [11754.094110] Key type .llcrypt registered [11754.211715] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11768.915558] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11770.034869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11770.300303] alg: No test for adler32 (adler32-zlib) [11771.541915] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11771.823714] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11773.535240] Key type lgssc registered [11774.844943] Lustre: Echo OBD driver; http://www.lustre.org/ [11800.647870] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:23:31 (1787804611) [11806.713708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11806.802426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.037718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11808.066181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11808.172102] Lustre: lustre-MDT0000: new disk, initializing [11808.229971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.242977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11812.220437] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11821.292041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.409532] Lustre: 326424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11821.444467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11821.447119] Lustre: Skipped 1 previous similar message [11821.490200] Lustre: lustre-MDT0001: new disk, initializing [11821.573372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.607943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11821.621438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11825.178820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11832.575085] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11837.278182] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11842.931633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11843.212132] Lustre: lustre-OST0000: new disk, initializing [11843.215813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11843.221421] Lustre: 328391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11843.330441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11849.467905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11853.381460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11853.400682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11853.556815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11858.743933] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11874.279940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.288743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11874.293087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11874.309127] LustreError: Skipped 1 previous similar message [11879.341737] Lustre: server umount lustre-OST0000 complete [11884.520664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11884.522018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11884.543789] Lustre: Skipped 2 previous similar messages [11884.561150] Lustre: Skipped 2 previous similar messages [11889.394071] Lustre: server umount lustre-MDT0000 complete [11892.995089] LustreError: 326417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804705 with bad export cookie 3991328841251581109 [11892.998734] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11893.001076] LustreError: 326417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11893.340508] Lustre: server umount lustre-MDT0001 complete [11904.596357] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11907.010158] Key type lgssc unregistered [11907.347951] LNet: 330302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11907.354902] LNetError: 330302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11907.370913] LNet: Removed LNI 192.168.202.119@tcp [11908.128543] Key type .llcrypt unregistered [11908.131262] Key type ._llcrypt unregistered [11930.329477] Key type ._llcrypt registered [11930.332100] Key type .llcrypt registered [11930.475616] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11931.816981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11931.856849] alg: No test for adler32 (adler32-zlib) [11933.221700] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11933.434172] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11935.071169] Key type lgssc registered [11936.068746] Lustre: Echo OBD driver; http://www.lustre.org/ [11943.834866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11943.856090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.580943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11949.778890] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11955.182719] LustreError: 331578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.209780] LustreError: 331578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11956.906529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.392831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11962.761088] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11971.667712] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11977.564236] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11983.660919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11983.929620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11989.772312] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11996.987457] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12010.156296] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12011.903885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12017.640351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.649359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12017.651264] Lustre: Skipped 1 previous similar message [12017.658144] Lustre: Skipped 2 previous similar messages [12020.252532] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [12022.616704] Lustre: server umount lustre-MDT0000 complete [12022.757656] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.784641] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12025.333320] LustreError: 334495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.354040] LustreError: 331559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804838 with bad export cookie 30274404165281741 [12026.364521] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12027.873608] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.874164] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.907230] Lustre: Skipped 1 previous similar message [12027.910769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12030.467913] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [12031.919950] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.946229] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12032.638910] Lustre: server umount lustre-MDT0001 complete [12036.058161] Lustre: server umount lustre-OST0000 complete [12043.755932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.688737] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12051.318312] Lustre: 335758:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [12053.610395] Lustre: server umount MGS complete [12061.960541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.397609] LustreError: 336330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.500111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12066.219464] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12074.679882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.028257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12080.164610] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12088.442745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12088.815632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12095.064287] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12102.878650] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12104.443944] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12110.190292] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12112.890392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [12117.476913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.480707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12117.476766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12117.491797] Lustre: Skipped 1 previous similar message [12117.521129] Lustre: Skipped 2 previous similar messages [12123.166818] Lustre: server umount lustre-OST0000 complete [12126.178128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12126.182603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12126.191333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12126.197076] Lustre: Skipped 2 previous similar messages [12132.497845] Lustre: server umount lustre-MDT0000 complete [12132.832089] LustreError: 336330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.837250] LustreError: 336330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12135.744320] LustreError: 336312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804948 with bad export cookie 30274404165282581 [12135.750756] LustreError: 336312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12135.756186] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.969455] Lustre: server umount lustre-MDT0001 complete [12146.224472] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12148.684842] Key type lgssc unregistered [12148.968718] LNet: 339767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12148.972798] LNetError: 339767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12148.987759] LNet: Removed LNI 192.168.202.119@tcp [12149.726358] Key type .llcrypt unregistered [12149.728687] Key type ._llcrypt unregistered [12165.456218] Key type ._llcrypt registered [12165.458295] Key type .llcrypt registered [12165.600940] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12179.742389] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12180.614110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12180.667134] alg: No test for adler32 (adler32-zlib) [12181.841793] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12182.043091] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12183.679203] Key type lgssc registered [12184.720392] Lustre: Echo OBD driver; http://www.lustre.org/ [12218.467988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12218.487343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.776528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12219.810718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12219.899477] Lustre: lustre-MDT0000: new disk, initializing [12219.986263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12220.012724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12224.407388] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12235.080570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12235.244222] Lustre: 342926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12235.318029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12235.323916] Lustre: Skipped 1 previous similar message [12235.402552] Lustre: lustre-MDT0001: new disk, initializing [12235.521818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12235.591461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12235.602971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12240.519487] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12250.111220] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12257.044903] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12265.158399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12265.430614] Lustre: lustre-OST0000: new disk, initializing [12265.434439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12265.442816] Lustre: 344898:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12265.538498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12272.291632] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12274.270612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12274.286994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12274.450048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12281.707563] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.378827] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12286.577560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12290.671669] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12290.875992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12295.139149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12295.142186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12295.150468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12295.156041] LustreError: Skipped 1 previous similar message [12295.166981] Lustre: Skipped 1 previous similar message [12298.875910] Lustre: server umount lustre-OST0000 complete [12302.307468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12302.316063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.335520] Lustre: Skipped 1 previous similar message [12302.340390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.377354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.390205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12308.302613] Lustre: server umount lustre-MDT0000 complete [12311.479582] LustreError: 342919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805123 with bad export cookie 9584161799267674921 [12311.483380] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.490724] LustreError: 342919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12311.763696] Lustre: server umount lustre-MDT0001 complete [12318.592588] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:32:09 (1787805129) [12331.946947] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12341.704607] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12378.272321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12378.529514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12378.559321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12378.628649] Lustre: lustre-MDT0000: new disk, initializing [12378.731569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12378.758904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12383.615142] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12393.563819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12393.643785] Lustre: 349655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12393.690242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12393.692437] Lustre: Skipped 1 previous similar message [12393.745406] Lustre: lustre-MDT0001: new disk, initializing [12393.802700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12393.844770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12393.867424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12398.462915] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12408.243738] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12413.523914] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12419.517121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12419.778589] Lustre: lustre-OST0000: new disk, initializing [12419.782335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12419.788198] Lustre: 351623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12419.909403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12426.635868] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12427.881673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12427.899266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12428.106390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12437.070888] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12466.687040] Lustre: server umount lustre-OST0000 complete [12470.752814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12470.760391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12470.771511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12470.774481] Lustre: Skipped 1 previous similar message [12479.457959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12479.461610] Lustre: Skipped 3 previous similar messages [12480.591993] Lustre: server umount lustre-MDT0000 complete [12484.293342] LustreError: 349650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805296 with bad export cookie 9584161799267675803 [12484.308970] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12484.315028] LustreError: 349650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12493.881708] Lustre: server umount lustre-MDT0001 complete [12505.704301] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12508.765841] Key type lgssc unregistered [12509.082595] LNet: 353780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12509.086590] LNetError: 353780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12509.100156] LNet: Removed LNI 192.168.202.119@tcp [12510.052986] Key type .llcrypt unregistered [12510.055045] Key type ._llcrypt unregistered [12599.233477] Key type ._llcrypt registered [12599.240554] Key type .llcrypt registered [12599.365346] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12599.596303] Key type .llcrypt unregistered [12599.599097] Key type ._llcrypt unregistered [12675.805854] Key type ._llcrypt registered [12675.810806] Key type .llcrypt registered [12676.022685] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12676.363565] Key type .llcrypt unregistered [12676.366548] Key type ._llcrypt unregistered [12706.236137] Key type ._llcrypt registered [12706.237710] Key type .llcrypt registered [12706.368985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12706.707256] Key type .llcrypt unregistered [12706.709424] Key type ._llcrypt unregistered [12734.564359] Key type ._llcrypt registered [12734.567155] Key type .llcrypt registered [12734.664473] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12734.894295] Key type .llcrypt unregistered [12734.895510] Key type ._llcrypt unregistered [12804.165642] Key type ._llcrypt registered [12804.167729] Key type .llcrypt registered [12804.259983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12804.628328] Key type .llcrypt unregistered [12804.630212] Key type ._llcrypt unregistered [12831.677730] Key type ._llcrypt registered [12831.679944] Key type .llcrypt registered [12831.809157] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12832.070264] Key type .llcrypt unregistered [12832.071872] Key type ._llcrypt unregistered [12858.821396] Key type ._llcrypt registered [12858.822883] Key type .llcrypt registered [12858.909623] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12859.226484] Key type .llcrypt unregistered [12859.228772] Key type ._llcrypt unregistered [12887.375433] Key type ._llcrypt registered [12887.384665] Key type .llcrypt registered [12887.494488] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12887.723584] Key type .llcrypt unregistered [12887.725398] Key type ._llcrypt unregistered [12916.836906] Key type ._llcrypt registered [12916.839427] Key type .llcrypt registered [12916.965542] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12917.261474] Key type .llcrypt unregistered [12917.273185] Key type ._llcrypt unregistered [12947.497992] Key type ._llcrypt registered [12947.499399] Key type .llcrypt registered [12947.592469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12947.819372] Key type .llcrypt unregistered [12947.823441] Key type ._llcrypt unregistered [13000.528380] Key type ._llcrypt registered [13000.530544] Key type .llcrypt registered [13000.626637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13001.133336] Key type .llcrypt unregistered [13001.135374] Key type ._llcrypt unregistered [13033.760814] Key type ._llcrypt registered [13033.763414] Key type .llcrypt registered [13033.957803] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13034.235659] Key type .llcrypt unregistered [13034.238315] Key type ._llcrypt unregistered [13058.803932] Key type ._llcrypt registered [13058.805491] Key type .llcrypt registered [13058.881478] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13059.152768] Key type .llcrypt unregistered [13059.158674] Key type ._llcrypt unregistered [13105.904588] Key type ._llcrypt registered [13105.907058] Key type .llcrypt registered [13106.049876] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13122.075250] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13123.841616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13124.030019] alg: No test for adler32 (adler32-zlib) [13125.265025] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [13125.510661] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [13127.224399] Key type lgssc registered [13128.394742] Lustre: Echo OBD driver; http://www.lustre.org/ [13169.781176] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13169.816432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13171.257204] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13171.274362] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13171.361232] Lustre: tst250fs-MDT0000: new disk, initializing [13171.432730] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13171.448216] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13176.337837] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13185.438779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13185.497246] Lustre: 374075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13185.531734] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13185.533810] Lustre: Skipped 1 previous similar message [13185.582483] Lustre: tst250fs-MDT0001: new disk, initializing [13185.656446] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13185.696735] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13185.705726] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13190.275465] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13194.546075] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13199.547678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13199.744447] Lustre: tst250fs-OST0000: new disk, initializing [13199.752352] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13199.758845] Lustre: 375701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13199.822525] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13203.545973] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13203.553486] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13203.693774] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13204.177989] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13211.672586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13211.837909] Lustre: tst250fs-OST0001: new disk, initializing [13211.840926] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13211.849247] Lustre: 376574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13211.936806] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13216.741835] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13218.878959] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13218.886458] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13218.950178] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13221.237628] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13223.446920] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13223.451964] Lustre: Skipped 2 previous similar messages [13241.830845] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13241.851183] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13241.870742] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13244.396262] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13244.403770] Lustre: Skipped 1 previous similar message [13244.404497] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13244.419471] Lustre: Skipped 3 previous similar messages [13245.854414] Lustre: server umount tst250fs-MDT0000 complete [13249.421394] LustreError: 374065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806061 with bad export cookie 7013061746442606115 [13249.425190] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13249.429179] LustreError: 374065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13249.515935] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13249.522734] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [13249.523655] Lustre: Skipped 2 previous similar messages [13249.532551] Lustre: Skipped 1 previous similar message [13249.705486] Lustre: server umount tst250fs-MDT0001 complete [13255.344866] Lustre: server umount tst250fs-OST0000 complete [13258.807880] Lustre: server umount tst250fs-OST0001 complete [13275.763941] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13283.775976] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13313.196569] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:48:44 (1787806124) [13314.549918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13333.048362] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13344.848863] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13381.069976] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13064 sec ========= 00:49:51 (1787806191) [13382.706361] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:49:53 (1787806193) === [13386.459503] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:49:57 (1787806197) === [13428.111612] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13431.060426] Key type lgssc unregistered [13431.381680] LNet: 385702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13431.394656] LNetError: 385702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13431.417813] LNet: Removed LNI 192.168.202.119@tcp [13432.162505] Key type .llcrypt unregistered [13432.167234] Key type ._llcrypt unregistered