[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461747182 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003093] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008396] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012131] LSM: Security Framework initializing [ 0.013046] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.014936] *** VALIDATE selinux *** [ 0.022767] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026728] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] *** VALIDATE tmpfs *** [ 0.031457] *** VALIDATE proc *** [ 0.033060] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043075] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.045182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054320] rcu: Hierarchical SRCU implementation. [ 0.056413] smp: Bringing up secondary CPUs ... [ 0.057614] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.063081] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137088] node 0 deferred pages initialised in 69ms [ 0.140111] devtmpfs: initialized [ 0.141235] x86/mm: Memory block size: 128MB [ 0.143190] gcov: version magic: 0x41383552 [ 0.145356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151352] pinctrl core: initialized pinctrl subsystem [ 0.152147] [ 0.152430] ************************************************************* [ 0.154009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156009] ** ** [ 0.157015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158008] ** ** [ 0.160008] ** This means that this kernel is built to expose internal ** [ 0.161008] ** IOMMU data structures, which may compromise security on ** [ 0.163007] ** your system. ** [ 0.164009] ** ** [ 0.165009] ** If you see this message and you are not debugging the ** [ 0.167008] ** kernel, report this immediately to your vendor! ** [ 0.168007] ** ** [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172016] ************************************************************* [ 0.174779] NET: Registered protocol family 16 [ 0.176354] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185037] cpuidle: using governor menu [ 0.186723] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188455] PCI: Using configuration type 1 for base access [ 0.190092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201078] cryptd: max_cpu_qlen set to 1000 [ 0.203167] ACPI: Added _OSI(Module Device) [ 0.204009] ACPI: Added _OSI(Processor Device) [ 0.205011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210605] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217623] ACPI: Interpreter enabled [ 0.218067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220015] ACPI: Using IOAPIC for interrupt routing [ 0.221095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234748] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246969] acpiphp: Slot [2] registered [ 0.247077] acpiphp: Slot [5] registered [ 0.247955] acpiphp: Slot [6] registered [ 0.249099] acpiphp: Slot [7] registered [ 0.251004] acpiphp: Slot [8] registered [ 0.251927] acpiphp: Slot [9] registered [ 0.252088] acpiphp: Slot [10] registered [ 0.253083] acpiphp: Slot [3] registered [ 0.255132] acpiphp: Slot [4] registered [ 0.257107] acpiphp: Slot [11] registered [ 0.258095] acpiphp: Slot [12] registered [ 0.260107] acpiphp: Slot [13] registered [ 0.261179] acpiphp: Slot [14] registered [ 0.263129] acpiphp: Slot [15] registered [ 0.265213] acpiphp: Slot [16] registered [ 0.267109] acpiphp: Slot [17] registered [ 0.268113] acpiphp: Slot [18] registered [ 0.270084] acpiphp: Slot [19] registered [ 0.271126] acpiphp: Slot [20] registered [ 0.273108] acpiphp: Slot [21] registered [ 0.275171] acpiphp: Slot [22] registered [ 0.276153] acpiphp: Slot [23] registered [ 0.278103] acpiphp: Slot [24] registered [ 0.279212] acpiphp: Slot [25] registered [ 0.281112] acpiphp: Slot [26] registered [ 0.283118] acpiphp: Slot [27] registered [ 0.284141] acpiphp: Slot [28] registered [ 0.286093] acpiphp: Slot [29] registered [ 0.287097] acpiphp: Slot [30] registered [ 0.289263] acpiphp: Slot [31] registered [ 0.290058] PCI host bridge to bus 0000:00 [ 0.292029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306293] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312389] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320848] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338075] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340655] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344595] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359784] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366239] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.376016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419594] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467118] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.476015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.517015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.531014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.537579] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.541017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.546009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.561019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.567902] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.570972] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.575016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.583925] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.590887] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.594275] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.596836] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.599382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.601150] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.604100] iommu: Default domain type: Passthrough [ 0.605000] SCSI subsystem initialized [ 0.605000] ACPI: bus type USB registered [ 0.605071] usbcore: registered new interface driver usbfs [ 0.606000] usbcore: registered new interface driver hub [ 0.607270] usbcore: registered new device driver usb [ 0.608127] pps_core: LinuxPPS API ver. 1 registered [ 0.610007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.612055] PTP clock support registered [ 0.613093] EDAC MC: Ver: 3.0.0 [ 0.615121] PCI: Using ACPI for IRQ routing [ 0.616668] NetLabel: Initializing [ 0.617007] NetLabel: domain hash size = 128 [ 0.617897] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619061] NetLabel: unlabeled traffic allowed by default [ 0.621002] vgaarb: loaded [ 0.622267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.623008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.630000] clocksource: Switched to clocksource kvm-clock [ 0.710296] VFS: Disk quotas dquot_6.6.0 [ 0.711370] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.712748] *** VALIDATE ramfs *** [ 0.713487] *** VALIDATE hugetlbfs *** [ 0.714362] pnp: PnP ACPI init [ 0.715962] pnp: PnP ACPI: found 6 devices [ 0.731202] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.733258] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.734586] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.736031] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.737456] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.738880] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.740626] NET: Registered protocol family 2 [ 0.742354] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.745639] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.747947] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751731] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.754246] TCP: Hash tables configured (established 65536 bind 65536) [ 0.755991] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.757862] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.759686] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.761978] NET: Registered protocol family 1 [ 0.763677] RPC: Registered named UNIX socket transport module. [ 0.765020] RPC: Registered udp transport module. [ 0.766245] RPC: Registered tcp transport module. [ 0.767500] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.769052] NET: Registered protocol family 44 [ 0.769961] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.771793] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.773544] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.775267] PCI: CLS 0 bytes, default 64 [ 0.776257] Unpacking initramfs... [ 2.043516] debug: unmapping init [mem 0xffff8f42bcc54000-0xffff8f42bffbffff] [ 2.047452] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.049892] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.053116] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.519886] Initialise system trusted keyrings [ 2.521273] Key type blacklist registered [ 2.522678] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.529639] zbud: loaded [ 2.532042] *** VALIDATE nfs *** [ 2.532881] *** VALIDATE nfs4 *** [ 2.534022] pstore: using deflate compression [ 2.536702] Platform Keyring initialized [ 2.620124] NET: Registered protocol family 38 [ 2.621619] Key type asymmetric registered [ 2.623129] Asymmetric key parser 'x509' registered [ 2.624810] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.627742] io scheduler mq-deadline registered [ 2.629636] io scheduler kyber registered [ 2.631363] io scheduler bfq registered [ 2.633507] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.636725] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.639796] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.642996] ACPI: Power Button [PWRF] [ 2.647306] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.654844] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.674549] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.682201] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.696142] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.725039] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.753242] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.757977] Non-volatile memory driver v1.3 [ 2.759815] Linux agpgart interface v0.103 [ 2.794764] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.797756] vda: detected capacity change from 0 to 76779520 [ 2.813648] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.817124] vdb: detected capacity change from 0 to 1073741824 [ 2.834025] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.836766] vdc: detected capacity change from 0 to 2621440000 [ 2.851374] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.854606] vdd: detected capacity change from 0 to 2621440000 [ 2.868486] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.871754] vde: detected capacity change from 0 to 4294967296 [ 2.885804] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.888105] vdf: detected capacity change from 0 to 4294967296 [ 2.893737] libphy: Fixed MDIO Bus: probed [ 2.899820] usbcore: registered new interface driver usbserial_generic [ 2.901549] usbserial: USB Serial support registered for generic [ 2.903191] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.906460] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.907766] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.909583] mousedev: PS/2 mouse device common for all mice [ 2.912067] rtc_cmos 00:05: RTC can wake from S4 [ 2.914574] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.916321] rtc_cmos 00:05: registered as rtc0 [ 2.920941] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.923831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.924355] intel_pstate: CPU model not supported [ 2.930088] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.932404] hid: raw HID events driver (C) Jiri Kosina [ 2.935453] usbcore: registered new interface driver usbhid [ 2.936583] usbhid: USB HID core driver [ 2.937537] drop_monitor: Initializing network drop monitor service [ 2.939216] Initializing XFRM netlink socket [ 2.941086] NET: Registered protocol family 10 [ 2.943346] Segment Routing with IPv6 [ 2.944911] NET: Registered protocol family 17 [ 2.947464] mpls_gso: MPLS GSO support [ 2.952709] RAS: Correctable Errors collector initialized. [ 2.954844] AVX version of gcm_enc/dec engaged. [ 2.956426] AES CTR mode by8 optimization enabled [ 3.023490] sched_clock: Marking stable (3023448563, 0)->(3894183385, -870734822) [ 3.027441] registered taskstats version 1 [ 3.029545] Loading compiled-in X.509 certificates [ 3.031959] zswap: loaded using pool lzo/zbud [ 3.052461] Key type big_key registered [ 3.065609] Key type encrypted registered [ 3.066639] ima: No TPM chip found, activating TPM-bypass! [ 3.067960] ima: Allocated hash algorithm: sha1 [ 3.069218] ima: No architecture policies found [ 3.070380] evm: Initialising EVM extended attributes: [ 3.071402] evm: security.selinux [ 3.072161] evm: security.ima [ 3.072762] evm: security.capability [ 3.073480] evm: HMAC attrs: 0x1 [ 3.075169] rtc_cmos 00:05: setting system clock to 2026-09-16 00:20:30 UTC (1789518030) [ 3.079063] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 3.081143] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 3.088118] Write protecting the kernel read-only data: 28672k [ 3.090165] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.091810] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 3.116845] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.123171] systemd[1]: Detected virtualization kvm. [ 3.124314] systemd[1]: Detected architecture x86-64. [ 3.125400] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.149479] systemd[1]: No hostname configured. [ 3.150612] systemd[1]: Set hostname to . [ 3.151799] random: systemd: uninitialized urandom read (16 bytes read) [ 3.153267] systemd[1]: Initializing machine ID from random generator. [ 3.284958] random: systemd: uninitialized urandom read (16 bytes read) [ 3.287078] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.292939] random: systemd: uninitialized urandom read (16 bytes read) [ 3.296100] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.301370] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.937096] device-mapper: uevent: version 1.0.3 [ 3.938795] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.639408] virtio_net virtio0 ens2: renamed from eth0 [ 4.673350] random: fast init done [ 4.696329] scsi host0: ata_piix [ 4.712284] scsi host1: ata_piix [ 4.713976] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.716442] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.216056] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.577779] random: crng init done [ 9.579072] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.069825] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.119849] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.345678] SELinux: Disabled at runtime. [ 11.398823] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.407581] systemd[1]: Detected virtualization kvm. [ 11.409475] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.869650] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.873315] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.878953] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.882899] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.886251] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.893034] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.896957] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ 11.959631] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.334267] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.697811] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.771580] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.842973] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.871450] EDAC sbridge: Ver: 1.1.2 [ 14.249901] Key type dns_resolver registered [ 14.553332] NFS: Registering the id_resolver key type [ 14.555448] Key type id_resolver registered [ 14.556993] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg257-server login: [ 61.701317] libcfs: loading out-of-tree module taints kernel. [ 61.763246] Key type ._llcrypt registered [ 61.766128] Key type .llcrypt registered [ 61.906824] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 81.212638] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 82.920133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.933093] alg: No test for adler32 (adler32-zlib) [ 84.383280] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 85.321485] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 87.215259] Key type lgssc registered [ 89.148627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.802017] hrtimer: interrupt took 6005241 ns [ 105.395100] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.085600] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 159.244773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.329287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.571712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.631195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.750960] Lustre: lustre-MDT0000: new disk, initializing [ 160.874843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.895100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.065575] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.684328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.783771] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.809877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.814736] Lustre: Skipped 1 previous similar message [ 179.881259] Lustre: lustre-MDT0001: new disk, initializing [ 180.022044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.062908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.084161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.583560] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.811262] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.059536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.404556] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 198.422972] Lustre: lustre-OST0000: new disk, initializing [ 198.428687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.434922] Lustre: 8463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.501743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.804144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.821098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.874340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 206.608835] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.969235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.086780] Lustre: lustre-OST0001: new disk, initializing [ 223.090361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.095734] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.173799] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.918498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.934254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.006139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.849081] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.811228] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.346359] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.095255] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.853507] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing yml_node [ 269.587642] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 271.868089] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 274.699849] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 276.346371] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 20:25:01 EDT 2026 [ 293.375645] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 294.838906] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.752187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.753830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.780938] Lustre: Skipped 2 previous similar messages [ 310.801430] Lustre: Skipped 3 previous similar messages [ 315.876031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.880986] Lustre: Skipped 3 previous similar messages [ 316.808738] Lustre: server umount lustre-MDT0000 complete [ 321.113320] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518348 with bad export cookie 18033946060084331530 [ 321.119368] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.127454] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.111977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.114173] LustreError: 6533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.115089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.121222] Lustre: Skipped 2 previous similar messages [ 326.128505] LustreError: 6533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 327.649958] Lustre: server umount lustre-MDT0001 complete [ 332.076965] Lustre: server umount lustre-OST0000 complete [ 336.965958] Lustre: server umount lustre-OST0001 complete [ 344.642731] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 353.403303] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 394.451267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.704731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.749420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.859850] Lustre: lustre-MDT0000: new disk, initializing [ 394.940940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.972806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.992794] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 415.436293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.562321] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.695307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.699930] Lustre: Skipped 1 previous similar message [ 415.804777] Lustre: lustre-MDT0001: new disk, initializing [ 416.012472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.069989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.093497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.442960] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 433.171889] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.355477] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.098199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 452.663873] Lustre: lustre-OST0000: new disk, initializing [ 452.676766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 452.695276] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 452.827689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.801645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 453.816644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 453.999506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 461.554477] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 474.542066] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.269274] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.614416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 487.543806] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 487.789144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.073540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.075567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.091875] LustreError: Skipped 1 previous similar message [ 495.094425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.094435] Lustre: Skipped 1 previous similar message [ 495.124894] Lustre: Skipped 1 previous similar message [ 496.900781] Lustre: server umount lustre-OST0000 complete [ 503.266234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 503.285707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.308355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.317243] Lustre: Skipped 1 previous similar message [ 507.788207] Lustre: server umount lustre-MDT0000 complete [ 510.434967] LustreError: 15969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.452348] LustreError: 15969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 512.448754] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518539 with bad export cookie 18033946060084333063 [ 512.453028] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.465232] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.989704] Lustre: server umount lustre-MDT0001 complete [ 519.840021] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:29:05 (1789518545) [ 541.680359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.296516] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.474289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.828167] LustreError: 19938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.226749] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 551.907423] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.600616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.770653] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 572.223668] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 578.838363] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 581.097188] Lustre: Failing over lustre-MDT0000 [ 581.107914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.122114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 581.124027] Lustre: Skipped 2 previous similar messages [ 581.138145] Lustre: Skipped 3 previous similar messages [ 581.369183] Lustre: server umount lustre-MDT0000 complete [ 583.213490] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.244393] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 599.530018] Lustre: 3657:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789518610/real 1789518610] req@ffff8f43375c0380 x1876445743974400/t0(0) o400->MGC192.168.202.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789518626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 599.590137] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.626300] LustreError: 19938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.657496] LustreError: 19938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 601.928315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.240142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.244396] Lustre: Skipped 1 previous similar message [ 610.318361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 615.397023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 615.412046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 615.457352] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 616.359313] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 628.760890] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 635.873109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.876060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.896555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 639.605626] Lustre: server umount lustre-MDT0000 complete [ 643.048902] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.095349] LustreError: 19937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 644.433710] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518671 with bad export cookie 18033946060084334477 [ 644.445596] LustreError: 19917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 644.455549] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.163724] Lustre: server umount lustre-MDT0001 complete [ 655.680540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.225089] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 672.928765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.798074] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 687.607865] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.371910] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.830525] Lustre: Failing over lustre-MDT0000 [ 697.261439] Lustre: server umount lustre-MDT0000 complete [ 697.897028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.917415] Lustre: Skipped 1 previous similar message [ 708.066316] LustreError: 24221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.087508] LustreError: 24221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 714.215352] Lustre: 3659:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789518725/real 1789518725] req@ffff8f433ce90700 x1876445744006400/t0(0) o400->MGC192.168.202.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789518741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 714.257967] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.721598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.473610] LustreError: 3656:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f432f0bca80 x1876445744009600/t0(0) o250->MGC192.168.202.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 725.810275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 730.689556] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 731.107488] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 731.113808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 731.127709] Lustre: Skipped 1 previous similar message [ 731.182053] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 742.541653] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 748.515024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.538965] Lustre: Skipped 1 previous similar message [ 748.551967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.560905] Lustre: Skipped 3 previous similar messages [ 753.749962] Lustre: server umount lustre-MDT0000 complete [ 758.667037] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518786 with bad export cookie 18033946060084335415 [ 758.672656] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.683126] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 759.127345] Lustre: server umount lustre-MDT0001 complete [ 770.395232] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:33:15 (1789518795) [ 790.112643] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 802.227792] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 850.938921] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 866.338556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.604882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 866.627834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 866.688387] Lustre: lustre-MDT0000: new disk, initializing [ 866.789656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.796778] Lustre: Skipped 3 previous similar messages [ 866.811819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 873.347896] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 887.055461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.177565] Lustre: 29658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 887.650540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 893.576811] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 906.051198] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 913.565112] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.691471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.077224] Lustre: 31630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 923.386806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 923.442534] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 923.593302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 923.612759] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 929.990946] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 940.917562] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 947.205937] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 974.821912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 974.823225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.839770] LustreError: Skipped 1 previous similar message [ 974.842809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.883499] Lustre: Skipped 2 previous similar messages [ 976.985210] Lustre: server umount lustre-OST0000 complete [ 988.222235] Lustre: server umount lustre-MDT0000 complete [ 990.188896] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.216800] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 992.766868] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519020 with bad export cookie 18033946060084335821 [ 992.775806] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.780996] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 993.451259] Lustre: server umount lustre-MDT0001 complete [ 1009.170610] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1013.221915] Key type lgssc unregistered [ 1013.508703] LNet: 33610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.515402] LNetError: 33610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.558578] LNet: Removed LNI 192.168.202.157@tcp [ 1015.076147] Key type .llcrypt unregistered [ 1015.077976] Key type ._llcrypt unregistered [ 1044.611295] Key type ._llcrypt registered [ 1044.614184] Key type .llcrypt registered [ 1044.793258] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1046.429728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.643430] alg: No test for adler32 (adler32-zlib) [ 1047.803937] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1048.204401] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1049.959492] Key type lgssc registered [ 1051.477798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.045835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.072865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.436992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.660736] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1071.590220] LustreError: 35023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.623087] LustreError: 35023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1076.707287] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.751347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.230020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.414595] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1095.977707] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1103.528761] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1111.703753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.092519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.987319] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1123.383929] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1128.053145] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1132.628994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1151.613333] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:39:36 (1789519176) [ 1153.693670] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1155.669424] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1162.920681] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:39:48 (1789519188) [ 1173.138363] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:39:58 (1789519198) [ 1174.308411] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1184.619974] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1184.633764] Lustre: Skipped 2 previous similar messages [ 1186.519833] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:40:11 (1789519211) [ 1188.644633] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1188.652214] Lustre: Skipped 2 previous similar messages [ 1197.464485] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1197.475848] Lustre: Skipped 4 previous similar messages [ 1214.331243] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:40:39 (1789519239) [ 1217.716256] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1222.531225] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1233.866440] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:40:58 (1789519258) [ 1236.409499] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1236.417609] Lustre: Skipped 6 previous similar messages [ 1253.156932] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:41:18 (1789519278) [ 1270.759127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1270.765415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.776037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1270.777043] Lustre: Skipped 1 previous similar message [ 1270.789046] Lustre: Skipped 1 previous similar message [ 1274.020360] Lustre: server umount lustre-OST0000 complete [ 1280.995958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.997322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.004020] Lustre: Skipped 1 previous similar message [ 1281.011950] Lustre: Skipped 1 previous similar message [ 1284.919098] Lustre: server umount lustre-MDT0000 complete [ 1286.113704] LustreError: 35751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.145124] LustreError: 35751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1289.836216] LustreError: 37968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519317 with bad export cookie 2015420730602366859 [ 1289.849977] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.853979] LustreError: 37968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1290.442164] Lustre: server umount lustre-MDT0001 complete [ 1305.058845] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1308.434431] Key type lgssc unregistered [ 1308.923740] LNet: 42905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.929961] LNetError: 42905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.959964] LNet: Removed LNI 192.168.202.157@tcp [ 1310.098268] Key type .llcrypt unregistered [ 1310.101158] Key type ._llcrypt unregistered [ 1331.383133] Key type ._llcrypt registered [ 1331.390710] Key type .llcrypt registered [ 1331.490699] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 1347.784054] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1349.361422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.475849] alg: No test for adler32 (adler32-zlib) [ 1350.683805] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1350.895703] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1352.631257] Key type lgssc registered [ 1353.996866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.387450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.454119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.764660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1395.794711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1395.893056] Lustre: lustre-MDT0000: new disk, initializing [ 1395.987568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.012350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1400.379670] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1412.300242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.454279] Lustre: 46065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1412.507621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1412.511131] Lustre: Skipped 1 previous similar message [ 1412.613948] Lustre: lustre-MDT0001: new disk, initializing [ 1412.713549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.744466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1412.762992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1417.629144] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1427.336290] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1433.625836] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1441.597314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.846480] Lustre: lustre-OST0000: new disk, initializing [ 1441.850538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1441.857310] Lustre: 48034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1441.943396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.465955] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1449.595545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1449.611608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1449.827498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1458.762992] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.641703] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1464.875927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1469.596235] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1469.773584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1475.552731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.558737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.573878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.595251] Lustre: Skipped 1 previous similar message [ 1480.690870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.700719] Lustre: Skipped 1 previous similar message [ 1485.795186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.804301] Lustre: Skipped 2 previous similar messages [ 1486.303219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.562698] Lustre: server umount lustre-OST0000 complete [ 1490.914923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.943251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.035901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.038706] Lustre: Skipped 2 previous similar messages [ 1497.009278] Lustre: server umount lustre-MDT0000 complete [ 1500.593269] LustreError: 46058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519528 with bad export cookie 11480021790664459889 [ 1500.602340] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.615313] LustreError: 46058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1500.967722] Lustre: server umount lustre-MDT0001 complete [ 1513.700140] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:45:38 (1789519538) [ 1522.066899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.857348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.290194] LustreError: 50495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.303181] LustreError: 50495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1529.568262] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1533.411972] LustreError: 50496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.534873] LustreError: 50495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.005161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.412426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1545.828952] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1555.237933] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1563.023842] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1571.806388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.071751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.588700] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1586.394515] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1601.187960] LustreError: 53047:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1602.074742] Lustre: Modifying parameter general.timeout=1 in log params [ 1603.082899] Lustre: Modifying parameter general.timeout=2 in log params [ 1605.328565] Lustre: Modifying parameter general.timeout=4 in log params [ 1605.332700] Lustre: Skipped 1 previous similar message [ 1610.303770] Lustre: Modifying parameter general.timeout=9 in log params [ 1610.307336] Lustre: Skipped 4 previous similar messages [ 1619.378111] Lustre: Modifying parameter general.timeout=18 in log params [ 1619.383041] Lustre: Skipped 8 previous similar messages [ 1636.225469] Lustre: Modifying parameter general.timeout=34 in log params [ 1636.228428] Lustre: Skipped 15 previous similar messages [ 1668.375261] Lustre: Modifying parameter general.timeout=66 in log params [ 1668.378519] Lustre: Skipped 31 previous similar messages [ 1735.379288] Lustre: Disabling parameter general.timeout= in log params [ 1735.390295] Lustre: Skipped 63 previous similar messages [ 1738.957251] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:49:24 (1789519764) [ 1741.866778] Lustre: 60008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1744.161686] Lustre: 60106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1748.419695] Lustre: 60303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1748.433691] Lustre: 60303:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1752.765308] Lustre: 60515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1752.780275] Lustre: 60515:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1782.856192] Lustre: 61878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1782.886559] LustreError: 61878:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1787.599367] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:50:12 (1789519812) [ 1804.777605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.803370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.804786] Lustre: Skipped 3 previous similar messages [ 1804.816452] Lustre: Skipped 2 previous similar messages [ 1809.893293] LustreError: 50495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.909533] LustreError: 50495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1809.972640] Lustre: server umount lustre-MDT0000 complete [ 1814.889214] LustreError: 53097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519842 with bad export cookie 11480021790664460771 [ 1814.893784] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.898867] LustreError: 53097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1815.008961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.010605] LustreError: 50490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.381854] Lustre: server umount lustre-MDT0001 complete [ 1822.394282] Lustre: server umount lustre-OST0000 complete [ 1854.857305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.117173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1855.323796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.404807] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1869.797741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.933655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1869.985100] Lustre: 64408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1875.540712] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1884.376386] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.649734] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.406718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.642205] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1898.951906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.959343] Lustre: Skipped 1 previous similar message [ 1905.566478] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1913.823883] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1915.533430] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1915.547865] Lustre: Skipped 23 previous similar messages [ 1915.593339] Lustre: 66978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1925.099432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1925.110492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.130564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1925.147106] Lustre: Skipped 1 previous similar message [ 1929.224724] Lustre: server umount lustre-OST0000 complete [ 1935.842222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.870356] Lustre: Skipped 1 previous similar message [ 1940.603490] Lustre: server umount lustre-MDT0000 complete [ 1940.963353] LustreError: 66846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.982232] LustreError: 66846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1945.183983] LustreError: 64402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519972 with bad export cookie 11480021790664463956 [ 1945.196240] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.204753] LustreError: 64402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1945.527332] Lustre: server umount lustre-MDT0001 complete [ 1958.956175] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1961.939353] Key type lgssc unregistered [ 1962.294973] LNet: 68186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.300199] LNetError: 68186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.320396] LNet: Removed LNI 192.168.202.157@tcp [ 1963.377212] Key type .llcrypt unregistered [ 1963.381380] Key type ._llcrypt unregistered [ 1993.695720] Key type ._llcrypt registered [ 1993.702477] Key type .llcrypt registered [ 1993.815481] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1994.602564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.626339] alg: No test for adler32 (adler32-zlib) [ 1995.915546] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1996.151251] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1997.800282] Key type lgssc registered [ 1999.094805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.526971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.544162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.346926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.242952] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2018.797698] LustreError: 69600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.819929] LustreError: 69600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2023.913495] LustreError: 69599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.545872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.416947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.450780] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2039.755629] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.567796] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2055.413099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.217773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.052330] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2073.329416] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2086.935449] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2096.611403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.620792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.635716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.169097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.173551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.186914] Lustre: Skipped 1 previous similar message [ 2099.208498] Lustre: Skipped 1 previous similar message [ 2101.888416] Lustre: server umount lustre-MDT0000 complete [ 2104.299922] LustreError: 69600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.152515] LustreError: 69579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520133 with bad export cookie 13909338440608883508 [ 2106.171532] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.592658] Lustre: server umount lustre-MDT0001 complete [ 2121.056083] Lustre: server umount lustre-OST0000 complete [ 2154.220412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.437812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2154.458315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2154.710341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.196158] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2170.007661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.299734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2170.371796] Lustre: 74473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2170.538245] LustreError: 74484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.563076] LustreError: 74484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2171.409343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.936180] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2185.470352] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.937513] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2203.608222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2203.919791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2204.505754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.776501] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2220.796860] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2222.728436] Lustre: 77049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2224.630491] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2231.776854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2231.785409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.815020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.822053] Lustre: Skipped 1 previous similar message [ 2236.173968] Lustre: server umount lustre-OST0000 complete [ 2242.528458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2242.529959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.534654] LustreError: Skipped 1 previous similar message [ 2242.535676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.535681] Lustre: Skipped 2 previous similar messages [ 2242.598438] Lustre: Skipped 2 previous similar messages [ 2247.002564] Lustre: server umount lustre-MDT0000 complete [ 2247.662407] LustreError: 74484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.677025] LustreError: 74484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2251.272989] LustreError: 74466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520278 with bad export cookie 13909338440608884348 [ 2251.273793] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.283782] LustreError: 74466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2251.734611] Lustre: server umount lustre-MDT0001 complete [ 2264.298473] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2267.510231] Key type lgssc unregistered [ 2267.845807] LNet: 78259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.850741] LNetError: 78259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.903758] LNet: Removed LNI 192.168.202.157@tcp [ 2269.719564] Key type .llcrypt unregistered [ 2269.722168] Key type ._llcrypt unregistered [ 2297.895235] Key type ._llcrypt registered [ 2297.899192] Key type .llcrypt registered [ 2298.092929] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2299.547915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.582572] alg: No test for adler32 (adler32-zlib) [ 2300.713690] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2301.148196] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2302.856331] Key type lgssc registered [ 2304.471142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.740973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.771137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.480902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.435415] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2325.995440] LustreError: 79674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.040195] LustreError: 79674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2331.112646] LustreError: 79673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.763720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.328286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2339.769564] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2349.491967] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2357.022076] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2366.929458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.283841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.475803] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2383.552192] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2394.134554] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2395.026851] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2396.052950] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2398.237444] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2398.243943] Lustre: Skipped 1 previous similar message [ 2402.709748] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2402.713918] Lustre: Skipped 3 previous similar messages [ 2411.397324] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2411.403760] Lustre: Skipped 7 previous similar messages [ 2428.081379] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2428.088433] Lustre: Skipped 17 previous similar messages [ 2460.821474] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2460.827214] Lustre: Skipped 30 previous similar messages [ 2512.935531] Lustre: 87800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2516.622745] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:02:21 (1789520541) [ 2518.654920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2520.837590] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:02:26 (1789520546) [ 2528.228670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.238060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.257709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.251256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.253375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.259811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.370890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.383487] Lustre: Skipped 2 previous similar messages [ 2539.504383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2539.521729] Lustre: Skipped 1 previous similar message [ 2539.999141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2540.102234] Lustre: server umount lustre-OST0000 complete [ 2549.219475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.228333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.242658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.111973] Lustre: server umount lustre-MDT0000 complete [ 2554.859627] LustreError: 81262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.885191] LustreError: 81262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2558.225974] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520585 with bad export cookie 6292443748215273256 [ 2558.237397] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.240677] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2558.886446] Lustre: server umount lustre-MDT0001 complete [ 2572.446846] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2575.585478] Key type lgssc unregistered [ 2576.227456] LNet: 89050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.238255] LNetError: 89050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.279358] LNet: Removed LNI 192.168.202.157@tcp [ 2577.370673] Key type .llcrypt unregistered [ 2577.373378] Key type ._llcrypt unregistered [ 2582.653606] Key type ._llcrypt registered [ 2582.655784] Key type .llcrypt registered [ 2582.770469] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2589.054980] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules [ 2589.816588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2589.926725] alg: No test for adler32 (adler32-zlib) [ 2591.094019] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2591.431875] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2593.151479] Key type lgssc registered [ 2594.526427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2596.521255] LustreError: 90297:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2619.284834] LustreError: 90297:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2619.382053] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2632.064683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2632.102198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.852766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.317696] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2644.456885] LustreError: 91325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.492469] LustreError: 91325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2649.365545] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:04:34 (1789520674) [ 2649.575755] LustreError: 91326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.561123] LustreError: 91357:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2654.572341] LustreError: 91357:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2654.582255] LustreError: 91357:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2654.916967] Lustre: server umount lustre-MDT0000 complete [ 2668.961870] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2671.987387] Key type lgssc unregistered [ 2672.250907] LNet: 92853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2672.257044] LNetError: 92853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2672.272550] LNet: Removed LNI 192.168.202.157@tcp [ 2673.371499] Key type .llcrypt unregistered [ 2673.373278] Key type ._llcrypt unregistered [ 2690.577646] Key type ._llcrypt registered [ 2690.581295] Key type .llcrypt registered [ 2690.741411] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2691.982825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2692.082550] alg: No test for adler32 (adler32-zlib) [ 2693.383429] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2693.743820] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2695.487220] Key type lgssc registered [ 2696.834273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2705.163731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2705.205390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.930430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.904878] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2717.156030] LustreError: 94066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.178981] LustreError: 94066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2722.280668] LustreError: 94067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.346116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.988554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.002652] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2734.653228] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2742.332202] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2751.825656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.979657] LustreError: 95893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.526073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.047802] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2767.952761] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2815.399996] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:07:20 (1789520840) [ 2817.402607] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2819.222938] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:07:24 (1789520844) [ 2823.137679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.152575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.161843] Lustre: Skipped 2 previous similar messages [ 2823.174191] Lustre: Skipped 2 previous similar messages [ 2828.262534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.270771] Lustre: Skipped 2 previous similar messages [ 2828.745036] Lustre: server umount lustre-MDT0000 complete [ 2832.432722] LustreError: 94047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520859 with bad export cookie 10706312971716531474 [ 2832.457208] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.771954] Lustre: server umount lustre-MDT0001 complete [ 2846.802437] Lustre: server umount lustre-OST0000 complete [ 2859.148750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.624445] LustreError: 98360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.844381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.354339] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2865.121401] LustreError: 98361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.489446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.276261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2879.350859] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2888.673565] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2895.382120] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2909.103417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.362175] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2909.366782] LustreError: 98349:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2909.388454] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2909.396705] LustreError: 100445:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2909.400704] LustreError: 100445:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2909.406739] LustreError: 100445:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2909.493043] Lustre: server umount lustre-OST0000 complete [ 2909.496292] LustreError: 100445:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2918.224809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.372720] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2918.384713] LustreError: 98350:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2918.423871] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2918.431906] LustreError: 100909:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2918.444643] LustreError: 100909:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2918.448581] LustreError: 100909:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2918.540499] Lustre: server umount lustre-OST0000 complete [ 2918.542355] LustreError: 100909:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2929.076117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.314087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2929.321360] Lustre: Found index 0 for lustre-OST0000, updating log [ 2929.335855] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2929.400853] Lustre: lustre-OST0000: new disk, initializing [ 2929.404562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2929.410213] Lustre: 101446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2929.792544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.166290] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2941.522285] Lustre: Failing over lustre-OST0000 [ 2941.811418] Lustre: server umount lustre-OST0000 complete [ 2950.114602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.129400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.130765] Lustre: Skipped 1 previous similar message [ 2950.157616] Lustre: Skipped 1 previous similar message [ 2955.235603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.238790] Lustre: Skipped 1 previous similar message [ 2960.355787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.355900] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2960.366865] Lustre: Skipped 1 previous similar message [ 2960.600376] Lustre: server umount lustre-MDT0000 complete [ 2965.344502] LustreError: 98340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520992 with bad export cookie 10706312971716534218 [ 2965.350298] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.354098] LustreError: 98340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2965.781529] Lustre: server umount lustre-MDT0001 complete [ 2975.454647] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:10:00 (1789521000) [ 2993.099065] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3005.142203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.763739] LustreError: 103795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.779127] LustreError: 103795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3005.849842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.054698] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3021.874635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.814626] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3032.720851] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3040.507507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.358367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.361374] Lustre: Skipped 1 previous similar message [ 3041.386538] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3047.525922] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3047.535488] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3047.551732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3049.039983] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3050.606631] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 3050.625663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3050.633369] Lustre: lustre-OST0000: Recovery over after 0:03, of 1 clients 1 recovered and 0 were evicted. [ 3050.634849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3050.645937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3060.802470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3061.012260] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3061.177326] Lustre: lustre-OST0001: new disk, initializing [ 3061.184511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3061.195872] Lustre: 106114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3070.095954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3070.109354] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3070.312636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3071.136344] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3079.674853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3085.711120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3090.923231] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.942395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3090.952679] Lustre: Skipped 1 previous similar message [ 3096.946604] Lustre: server umount lustre-MDT0001 complete [ 3100.013346] LustreError: 106594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.036613] LustreError: 106594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3108.560515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.795736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3108.797930] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3108.911153] Lustre: 103783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3109.858026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.874890] Lustre: Skipped 1 previous similar message [ 3110.488451] LustreError: 107653:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3110.498817] LustreError: 107653:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3110.514576] LustreError: 103787:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3110.665868] LustreError: 107658:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.157@tcp: cfg command failed: rc = -17 [ 3110.677596] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3110.677596] [ 3114.992352] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3115.293984] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3128.940898] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:12:33 (1789521153) [ 3151.246632] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3173.173120] Lustre: 109733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3198.527674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3202.783508] Lustre: 110867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3293.664828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3293.677567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.688722] Lustre: Skipped 1 previous similar message [ 3293.692146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.694924] Lustre: Skipped 6 previous similar messages [ 3297.061338] Lustre: server umount lustre-MDT0000 complete [ 3299.298804] LustreError: 103790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.316570] LustreError: 103790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3301.472633] LustreError: 103775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521328 with bad export cookie 10706312971716535219 [ 3301.474519] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.488081] LustreError: 103775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3301.810373] Lustre: server umount lustre-MDT0001 complete [ 3316.797840] Lustre: server umount lustre-OST0000 complete [ 3319.777239] Lustre: 93379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789521331/real 1789521331] req@ffff8f4201aa0000 x1876448479907968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789521347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3319.809959] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.825436] Lustre: Skipped 3 previous similar messages [ 3320.969270] Lustre: server umount lustre-OST0001 complete [ 3326.953482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3336.487786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3347.652282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3359.177466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3383.749965] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3395.287863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.318389] Lustre: lustre-MDT0000: reset Object Index mappings [ 3396.042691] LustreError: 114817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.182195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.028318] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3412.441253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.483274] Lustre: lustre-MDT0001: reset Object Index mappings [ 3418.018096] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3421.626936] Lustre: 115966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3429.639252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.285131] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3437.936244] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3440.972128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3444.622537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.876170] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3451.009514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3451.966920] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3463.473760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3468.499661] Lustre: 117844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3522.056775] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:19:07 (1789521547) [ 3548.129808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.143184] Lustre: Skipped 2 previous similar messages [ 3548.149253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.160884] Lustre: Skipped 7 previous similar messages [ 3554.001423] Lustre: server umount lustre-MDT0000 complete [ 3557.570990] LustreError: 114799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521584 with bad export cookie 10706312971716582315 [ 3557.574806] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.599710] LustreError: 114799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3558.369030] LustreError: 114812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.379960] LustreError: 114812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3564.234887] Lustre: server umount lustre-MDT0001 complete [ 3574.562776] Lustre: server umount lustre-OST0000 complete [ 3592.168732] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3592.507241] Lustre: server umount lustre-OST0001 complete [ 3600.210539] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 3612.968183] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3657.890851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.122388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3658.141954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3658.232545] Lustre: lustre-MDT0000: new disk, initializing [ 3658.310057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.312802] Lustre: Skipped 3 previous similar messages [ 3658.330205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3663.909832] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3672.168907] Lustre: server umount lustre-MDT0000 complete [ 3683.786570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.102184] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3684.117218] Lustre: 122120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.127263] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3688.207532] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3694.305504] Lustre: server umount lustre-MDT0000 complete [ 3708.818621] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:22:13 (1789521733) [ 3712.408495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3714.476439] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:22:19 (1789521739) [ 3734.539423] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3745.580653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.637714] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3760.520634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.642934] Lustre: 124171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3760.700182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3760.703228] Lustre: Skipped 1 previous similar message [ 3760.752855] Lustre: lustre-MDT0001: new disk, initializing [ 3760.825290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3760.835058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3767.197966] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3773.591640] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3782.054687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3782.384093] Lustre: lustre-OST0000: new disk, initializing [ 3782.387349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3782.403035] Lustre: 125749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3784.367919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3784.376844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3784.534401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.303909] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3801.687500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.813794] Lustre: lustre-OST0001: new disk, initializing [ 3801.816733] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3801.822828] Lustre: 126619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3803.432531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3803.452169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3803.612411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3808.894653] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3818.757488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3822.658798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4214.538270] Lustre: 133036:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4214.549933] Lustre: 133036:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4214.555547] Lustre: 133036:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4214.569238] Lustre: 133036:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4214.585252] Lustre: 133036:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4214.595403] Lustre: 133036:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4511.599505] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:35:36 (1789522536) [ 4528.618103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.628445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4528.636662] Lustre: Skipped 4 previous similar messages [ 4528.656881] Lustre: Skipped 11 previous similar messages [ 4532.522655] Lustre: server umount lustre-MDT0000 complete [ 4533.733066] LustreError: 125743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.767605] LustreError: 125743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4538.069898] LustreError: 124163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522565 with bad export cookie 10706312971716650761 [ 4538.087990] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.091305] LustreError: 124163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4538.849967] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.856351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4538.863547] Lustre: Skipped 3 previous similar messages [ 4538.888577] Lustre: Skipped 1 previous similar message [ 4544.919275] Lustre: server umount lustre-MDT0001 complete [ 4556.247806] Lustre: server umount lustre-OST0000 complete [ 4566.520704] Lustre: server umount lustre-OST0001 complete [ 4574.327573] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 4584.806567] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 4624.429293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.668532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4624.692133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4624.761313] Lustre: lustre-MDT0000: new disk, initializing [ 4624.829858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.837968] Lustre: Skipped 5 previous similar messages [ 4624.850954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.726268] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4641.582681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.712109] Lustre: 136102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4641.776983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4641.779298] Lustre: Skipped 1 previous similar message [ 4641.887349] Lustre: lustre-MDT0001: new disk, initializing [ 4642.057264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4642.071256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4646.503965] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4659.244446] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4666.174610] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4674.085914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.370693] Lustre: lustre-OST0000: new disk, initializing [ 4674.373941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4674.377755] Lustre: 138072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4676.448161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4676.469196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4676.613973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4681.452806] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4691.228600] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.752704] Lustre: lustre-MDD0000: changelog on [ 4708.460166] Lustre: lustre-MDD0001: changelog on [ 6274.195076] Lustre: lustre-MDD0001: changelog off [ 6275.052957] Lustre: 138445:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6275.429088] Lustre: lustre-MDD0000: changelog off [ 6279.138162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.138935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.141083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.159930] Lustre: Skipped 3 previous similar messages [ 6283.698987] Lustre: server umount lustre-OST0000 complete [ 6285.793213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.796226] LustreError: Skipped 1 previous similar message [ 6285.801460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.809096] Lustre: Skipped 1 previous similar message [ 6285.812043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.377909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.808306] Lustre: server umount lustre-MDT0000 complete [ 6293.833940] LustreError: 136095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524321 with bad export cookie 10706312971717468634 [ 6293.837720] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6293.841547] LustreError: 136095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6294.084954] Lustre: server umount lustre-MDT0001 complete [ 6299.133706] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6300.696051] Key type lgssc unregistered [ 6300.893706] LNet: 147168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6300.897422] LNetError: 147168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6300.905996] LNet: Removed LNI 192.168.202.157@tcp [ 6301.388083] Key type .llcrypt unregistered [ 6301.389706] Key type ._llcrypt unregistered [ 6311.746702] Key type ._llcrypt registered [ 6311.748297] Key type .llcrypt registered [ 6311.799913] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6319.965981] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6320.318873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6320.326390] alg: No test for adler32 (adler32-zlib) [ 6321.292960] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6321.429665] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6323.039790] Key type lgssc registered [ 6323.633442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6342.799143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6342.816445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.936875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6343.957686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6344.003424] Lustre: lustre-MDT0000: new disk, initializing [ 6344.061911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.071902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6346.441665] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6352.475308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.533608] Lustre: 150340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6352.573477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6352.576839] Lustre: Skipped 1 previous similar message [ 6352.621986] Lustre: lustre-MDT0001: new disk, initializing [ 6352.678825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.697621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6352.706760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6355.198951] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6360.237346] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6362.639428] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6366.170525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.318515] Lustre: lustre-OST0000: new disk, initializing [ 6366.320240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6366.323262] Lustre: 152306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6366.364103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.640361] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6374.762569] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.956116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6375.966524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6376.013770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6383.042452] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_module obdecho/obdecho [ 6383.909524] LustreError: 153308:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6383.914266] LustreError: 153308:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6384.804741] LustreError: 153405:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6391.266951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.268190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.274381] Lustre: Skipped 2 previous similar messages [ 6391.276821] Lustre: Skipped 1 previous similar message [ 6396.261907] Lustre: server umount lustre-MDT0000 complete [ 6397.882372] LustreError: 150333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524425 with bad export cookie 4748012648939662663 [ 6397.885606] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.887947] LustreError: 150333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6398.029413] Lustre: server umount lustre-MDT0001 complete [ 6410.089778] Lustre: server umount lustre-OST0000 complete [ 6420.232579] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6425.356369] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6444.029443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.146027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6444.164522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6444.200507] Lustre: lustre-MDT0000: new disk, initializing [ 6444.241500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6444.248824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6446.210434] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6451.672019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.719052] Lustre: 157113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6451.751307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6451.754213] Lustre: Skipped 1 previous similar message [ 6451.794641] Lustre: lustre-MDT0001: new disk, initializing [ 6451.841488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6451.859227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6451.866096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6453.701886] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6458.227784] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6460.657527] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6464.002161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.128211] Lustre: lustre-OST0000: new disk, initializing [ 6464.130384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6464.133330] Lustre: 159080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6464.170384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.733098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6465.737528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6465.778815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6466.635512] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6471.455032] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6473.602846] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.730816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6475.689235] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6475.785535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6481.378418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6481.379824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.381238] LustreError: Skipped 1 previous similar message [ 6481.386589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6481.391468] Lustre: Skipped 2 previous similar messages [ 6482.718869] Lustre: server umount lustre-OST0000 complete [ 6486.496742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.497715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.501463] Lustre: Skipped 2 previous similar messages [ 6490.529503] Lustre: server umount lustre-MDT0000 complete [ 6491.617863] LustreError: 157126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.625723] LustreError: 157126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6492.111258] LustreError: 157910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524519 with bad export cookie 4748012648939663622 [ 6492.114301] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.115776] LustreError: 157910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6492.239714] Lustre: server umount lustre-MDT0001 complete [ 6494.013170] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:08:40 (1789524520) [ 6494.812793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6495.785185] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:08:41 (1789524521) [ 6499.568540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.812614] LustreError: 161492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.854079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.840838] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6504.929192] LustreError: 161493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.659141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.479763] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6510.452909] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6512.674298] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6515.466370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6518.064604] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6521.214217] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6530.017842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.024565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6530.026171] Lustre: Skipped 2 previous similar messages [ 6530.027946] Lustre: Skipped 2 previous similar messages [ 6535.136285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6535.144668] Lustre: Skipped 3 previous similar messages [ 6536.183085] Lustre: server umount lustre-MDT0000 complete [ 6537.051570] LustreError: 161489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.909532] LustreError: 161474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524565 with bad export cookie 4748012648939664497 [ 6537.911729] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.915930] LustreError: 161474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6540.256714] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.172310] LustreError: 161488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.182978] LustreError: 161488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6544.143163] Lustre: server umount lustre-MDT0001 complete [ 6547.193483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6552.745566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.852690] LustreError: 165328:0:(llog.c:1655:llog_backup()) MGC192.168.202.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6552.857836] Lustre: 165328:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6562.649176] LustreError: 165333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.654770] LustreError: 165333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6582.239926] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6582.247093] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6613.983293] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6613.986918] Lustre: 165328:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6617.949760] LustreError: 165351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.963227] LustreError: 165351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6654.445337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.448262] Lustre: Skipped 2 previous similar messages [ 6656.345870] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6658.534238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6660.646091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.459355] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.465790] LustreError: 165334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 46 previous similar messages [ 6683.103741] LustreError: 148563:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d6fc40e3 x1876452282838400/t0(0) o250->MGC192.168.202.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6683.120197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6684.967501] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6688.326929] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6688.741790] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6689.187071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6692.040101] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6692.773509] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6704.100106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.102208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6704.102742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.102747] Lustre: Skipped 3 previous similar messages [ 6704.110016] Lustre: Skipped 1 previous similar message [ 6704.286885] Lustre: server umount lustre-MDT0000 complete [ 6706.091781] LustreError: 164271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524733 with bad export cookie 4748012648939670979 [ 6706.094578] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.100546] LustreError: 164271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6706.243211] Lustre: server umount lustre-MDT0001 complete [ 6718.442450] Lustre: server umount lustre-OST0000 complete [ 6727.448431] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6731.665427] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6748.378237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.516483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6748.529649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6748.569284] Lustre: lustre-MDT0000: new disk, initializing [ 6748.610247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.613899] Lustre: Skipped 1 previous similar message [ 6748.624126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6750.437802] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6755.448472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.501996] Lustre: 170404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6755.608359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6757.183535] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6761.894017] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6764.381116] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.277205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.404526] Lustre: 172373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6769.892408] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6774.354732] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6775.313411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6775.358870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6776.149347] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6776.225441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6777.948996] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6778.029523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6780.385992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6780.386944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.389385] LustreError: Skipped 1 previous similar message [ 6780.393682] Lustre: Skipped 1 previous similar message [ 6780.396162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6780.399094] Lustre: Skipped 3 previous similar messages [ 6784.987278] Lustre: server umount lustre-OST0000 complete [ 6786.531072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6792.615201] Lustre: server umount lustre-MDT0000 complete [ 6794.091174] LustreError: 170399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524821 with bad export cookie 4748012648939687772 [ 6794.095619] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6794.099050] LustreError: 170399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6794.230340] Lustre: server umount lustre-MDT0001 complete [ 6797.762182] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:13:43 (1789524823) [ 6800.683436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.519130] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6806.164299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.887057] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6810.717980] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6812.830538] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.476362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.884361] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6820.781442] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.844603] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:14:17 (1789524857) [ 6836.194521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.197113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.198995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.203327] Lustre: Skipped 3 previous similar messages [ 6836.209188] Lustre: Skipped 4 previous similar messages [ 6838.240300] Lustre: server umount lustre-OST0000 complete [ 6854.111118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6854.178431] Lustre: server umount lustre-MDT0000 complete [ 6855.649250] LustreError: 175513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.655979] LustreError: 175513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6855.869955] LustreError: 174765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524883 with bad export cookie 4748012648939688640 [ 6855.876195] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6855.877135] LustreError: 174765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6856.051577] Lustre: server umount lustre-MDT0001 complete [ 6860.213565] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6861.278617] Key type lgssc unregistered [ 6861.411736] LNet: 178508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6861.414612] LNetError: 178508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6861.424301] LNet: Removed LNI 192.168.202.157@tcp [ 6861.766445] Key type .llcrypt unregistered [ 6861.767635] Key type ._llcrypt unregistered [ 6870.148584] Key type ._llcrypt registered [ 6870.150080] Key type .llcrypt registered [ 6870.190285] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6870.518965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6870.527129] alg: No test for adler32 (adler32-zlib) [ 6871.391441] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6871.488872] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6873.071205] Key type lgssc registered [ 6873.473086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6876.358321] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.364336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.807396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6896.927301] LustreError: 179739:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 6902.047388] LustreError: 179739:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6902.052546] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6902.055260] LustreError: MGC192.168.202.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6902.059727] LustreError: 179739:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6902.064201] LustreError: 179739:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6902.067596] LustreError: 179739:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6902.091355] Lustre: server umount lustre-OST0000 complete [ 6902.093158] LustreError: 179739:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6905.549685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.697110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.345491] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6911.627234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.785890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.271818] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6915.939096] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6917.968683] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6920.433037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.518497] LustreError: 182048:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6920.522857] Lustre: 182048:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6920.579515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.683101] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6925.282578] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6926.068132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.070609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.073524] Lustre: Skipped 1 previous similar message [ 6931.424296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.426734] Lustre: Skipped 4 previous similar messages [ 6932.250139] Lustre: server umount lustre-MDT0000 complete [ 6933.720130] LustreError: 180218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524961 with bad export cookie 5259680434226509280 [ 6933.722693] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.724187] LustreError: 180218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6933.865268] Lustre: server umount lustre-MDT0001 complete [ 6945.625212] Lustre: server umount lustre-OST0000 complete [ 6948.826531] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6952.610462] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6966.017643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.103812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6966.113831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6966.148600] Lustre: lustre-MDT0000: new disk, initializing [ 6966.169898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.175612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6967.764770] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6972.434457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.478724] Lustre: 185571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6972.500472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6972.503034] Lustre: Skipped 1 previous similar message [ 6972.537877] Lustre: lustre-MDT0001: new disk, initializing [ 6972.570167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6972.583238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6972.586833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6973.948027] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6977.995620] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6980.032075] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6982.514145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.611173] Lustre: lustre-OST0000: new disk, initializing [ 6982.613405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6982.617460] Lustre: 187534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6982.645853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.503308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6984.505874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6984.538117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6984.731788] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6988.613285] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6990.176200] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6990.254222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6991.848486] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6991.928557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6994.912253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.914400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.915571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6994.918378] Lustre: Skipped 2 previous similar messages [ 6998.679497] Lustre: server umount lustre-OST0000 complete [ 7003.615788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.618475] LustreError: Skipped 1 previous similar message [ 7003.620535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.626071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.628332] Lustre: Skipped 1 previous similar message [ 7006.439273] Lustre: server umount lustre-MDT0000 complete [ 7007.793403] LustreError: 185562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525035 with bad export cookie 5259680434226509980 [ 7007.796206] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.797968] LustreError: 185562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7007.930903] Lustre: server umount lustre-MDT0001 complete [ 7011.628796] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:17:17 (1789525037) [ 7018.379375] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7019.468770] Key type lgssc unregistered [ 7019.602900] LNet: 190473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7019.605960] LNetError: 190473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7019.613389] LNet: Removed LNI 192.168.202.157@tcp [ 7019.948852] Key type .llcrypt unregistered [ 7019.950148] Key type ._llcrypt unregistered [ 7028.516281] Key type ._llcrypt registered [ 7028.517829] Key type .llcrypt registered [ 7028.555493] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7028.882884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7028.929240] alg: No test for adler32 (adler32-zlib) [ 7029.804871] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7029.912666] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7031.487282] Key type lgssc registered [ 7031.916591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7034.851872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7034.864129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.027175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.487658] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7040.481749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.595658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7042.019877] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7044.714905] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.811731] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7049.868345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.974244] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7049.976430] LustreError: 193526:0:(llog.c:1696:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7049.980020] Lustre: 193526:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7050.027114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.419242] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7055.058609] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.840069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7055.843496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.849643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.450309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.450605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.453678] Lustre: Skipped 1 previous similar message [ 7060.464023] Lustre: Skipped 1 previous similar message [ 7065.568991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7065.574618] Lustre: Skipped 3 previous similar messages [ 7070.175104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7070.238538] Lustre: server umount lustre-MDT0000 complete [ 7070.689408] LustreError: 191667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.695052] LustreError: 191667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7071.740110] LustreError: 191648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525099 with bad export cookie 3554708323828559200 [ 7071.742554] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7071.744557] LustreError: 191648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7073.759365] LustreError: 192394:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002a33fe65 x1876453026189952/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7073.841261] Lustre: server umount lustre-MDT0001 complete [ 7084.778730] Lustre: server umount lustre-OST0000 complete [ 7088.543296] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7092.426306] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7105.049738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.127588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7105.137762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7105.172652] Lustre: lustre-MDT0000: new disk, initializing [ 7105.194329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.202847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7106.552960] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7110.755305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7110.797474] Lustre: 197048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7110.820896] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7110.822627] Lustre: Skipped 1 previous similar message [ 7110.824319] LustreError: 197829:0:(llog.c:1696:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7110.827765] LustreError: 197829:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7110.829952] Lustre: 197829:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7110.840573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7110.842646] Lustre: Skipped 1 previous similar message [ 7110.881790] Lustre: lustre-MDT0001: new disk, initializing [ 7110.918316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7110.931809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7110.935043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7112.416944] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7116.533804] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7118.705629] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7121.460939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.556108] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7121.557741] Lustre: Skipped 1 previous similar message [ 7121.559782] LustreError: 198994:0:(llog.c:1696:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7121.564149] LustreError: 198994:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7121.566857] Lustre: 198994:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7121.571017] Lustre: 198994:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7121.580354] Lustre: lustre-OST0000: new disk, initializing [ 7121.582707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7121.585870] Lustre: 199017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.619073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.740459] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7127.034252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7127.038098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7127.077048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7127.677827] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7129.272072] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7129.350276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7131.022536] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7131.120939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7132.129449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7132.129660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7132.134136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7132.135710] LustreError: Skipped 1 previous similar message [ 7133.971537] Lustre: server umount lustre-OST0000 complete [ 7136.735948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.738824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.743775] Lustre: Skipped 1 previous similar message [ 7136.746078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.749179] Lustre: Skipped 1 previous similar message [ 7141.468900] Lustre: server umount lustre-MDT0000 complete [ 7142.368154] LustreError: 197058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.375058] LustreError: 197058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7142.868506] LustreError: 197040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525170 with bad export cookie 3554708323828559907 [ 7142.871103] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.873250] LustreError: 197040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7143.006302] Lustre: server umount lustre-MDT0001 complete [ 7146.669458] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:19:32 (1789525172) [ 7153.185959] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7157.173361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.314100] LustreError: 202136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.332348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.824483] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7161.962757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.597172] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7164.653654] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7166.959743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.175661] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7172.587264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7172.654273] Lustre: lustre-OST0001: new disk, initializing [ 7172.656634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7172.659048] Lustre: 204441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7173.574264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7173.577755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7173.618422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7174.867376] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7180.639590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7188.447696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7189.069807] Lustre: DEBUG MARKER: ADD OST3 [ 7190.719632] loop: module loaded [ 7190.751618] loop0: detected capacity change from 0 to 204800000 [ 7190.808523] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7190.811834] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7190.816594] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7192.926288] loop0: detected capacity change from 0 to 204800000 [ 7194.879660] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.949956] Lustre: lustre-OST0003: new disk, initializing [ 7194.951808] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7194.953720] Lustre: 206173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7194.985328] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7194.987982] Lustre: Skipped 3 previous similar messages [ 7195.798520] Lustre: DEBUG MARKER: STOP OST3 [ 7196.494377] Lustre: Failing over lustre-OST0003 [ 7196.538859] Lustre: server umount lustre-OST0003 complete [ 7196.564521] LustreError: 203622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.143510] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7201.126875] LustreError: 204579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.132731] LustreError: 204579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7201.167633] Lustre: DEBUG MARKER: START OST3 again [ 7203.561230] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.652459] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7205.152105] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7205.160429] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7205.711578] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7208.938025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7208.942114] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7208.951457] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7219.625543] Lustre: Failing over lustre-OST0003 [ 7219.674976] Lustre: server umount lustre-OST0003 complete [ 7224.293727] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7224.294302] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.295859] LustreError: Skipped 1 previous similar message [ 7224.299680] Lustre: Skipped 1 previous similar message [ 7224.302241] LustreError: 204579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.308242] LustreError: 204579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7234.016103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7234.018577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.023289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.026412] Lustre: Skipped 2 previous similar messages [ 7238.427194] Lustre: server umount lustre-MDT0000 complete [ 7239.669194] LustreError: 202118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525267 with bad export cookie 3554708323828560775 [ 7239.672777] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7239.673647] LustreError: 202118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7244.768452] LustreError: 202133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.773420] LustreError: 202133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7245.868352] Lustre: server umount lustre-MDT0001 complete [ 7247.154964] Lustre: server umount lustre-OST0000 complete [ 7248.483592] Lustre: server umount lustre-OST0001 complete [ 7250.628884] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7253.296687] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7266.279685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.405850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7266.417729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7266.453479] Lustre: lustre-MDT0000: new disk, initializing [ 7266.485188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.488457] Lustre: Skipped 1 previous similar message [ 7266.497449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7268.043585] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7272.681761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.731889] Lustre: 211583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7272.868166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7274.488534] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7278.586898] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7280.643558] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7283.110323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7283.214773] Lustre: lustre-OST0000: new disk, initializing [ 7283.216916] Lustre: Skipped 1 previous similar message [ 7283.218850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7283.221732] Lustre: Skipped 2 previous similar messages [ 7283.224517] Lustre: 213549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.794055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7284.796570] Lustre: Skipped 1 previous similar message [ 7284.798906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7284.831069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7285.269907] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7289.017539] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.411787] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7290.490626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7291.875795] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7291.945015] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7294.944871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.946913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.947821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7294.950362] Lustre: Skipped 6 previous similar messages [ 7294.954658] Lustre: Skipped 6 previous similar messages [ 7298.706785] Lustre: server umount lustre-OST0000 complete [ 7314.399116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7314.470850] Lustre: server umount lustre-MDT0000 complete [ 7315.424180] LustreError: 211596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.429293] LustreError: 211596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7315.834533] LustreError: 212378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525343 with bad export cookie 3554708323828563015 [ 7315.838344] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.838873] LustreError: 212378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7315.975754] Lustre: server umount lustre-MDT0001 complete [ 7317.349455] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:22:23 (1789525343) [ 7323.043195] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7326.608431] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7339.223670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.333646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7339.347464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7339.389864] Lustre: lustre-MDT0000: new disk, initializing [ 7339.420879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7340.844661] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7345.010163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.050065] Lustre: 218150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.153823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7346.593486] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7350.500992] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7352.552270] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7355.019713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.148215] Lustre: 220117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7356.896486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7357.239103] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7361.235438] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7362.835121] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7362.921460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7364.416168] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7364.497085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7367.136753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7367.139478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.140092] LustreError: Skipped 2 previous similar messages [ 7367.141125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7367.141129] Lustre: Skipped 6 previous similar messages [ 7367.144300] Lustre: Skipped 3 previous similar messages [ 7371.413310] Lustre: server umount lustre-OST0000 complete [ 7387.103121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7387.162056] Lustre: server umount lustre-MDT0000 complete [ 7388.485659] LustreError: 218145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525415 with bad export cookie 3554708323828563897 [ 7388.487953] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.490798] LustreError: 218145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7388.616035] Lustre: server umount lustre-MDT0001 complete [ 7391.471719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.621489] LustreError: 222331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7392.990214] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7395.694387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.801247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7395.802724] Lustre: Skipped 6 previous similar messages [ 7397.067823] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7399.473523] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7401.332069] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7403.645770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.637147] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7407.957619] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7426.528188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7426.530705] LustreError: Skipped 1 previous similar message [ 7430.800346] Lustre: server umount lustre-OST0000 complete [ 7432.672496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.673392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.676715] Lustre: Skipped 5 previous similar messages [ 7432.678187] Lustre: Skipped 11 previous similar messages [ 7438.233849] Lustre: server umount lustre-MDT0000 complete [ 7439.549601] LustreError: 223067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525466 with bad export cookie 3554708323828564779 [ 7439.553379] LustreError: 223067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7439.553752] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.669091] Lustre: server umount lustre-MDT0001 complete [ 7539.645389] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7540.681907] Key type lgssc unregistered [ 7540.819439] LNet: 226012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.821661] LNetError: 226012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7540.833377] LNet: Removed LNI 192.168.202.157@tcp [ 7541.171209] Key type .llcrypt unregistered [ 7541.172717] Key type ._llcrypt unregistered [ 7548.866810] Key type ._llcrypt registered [ 7548.868356] Key type .llcrypt registered [ 7548.906181] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7554.994026] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7555.410784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7555.418138] alg: No test for adler32 (adler32-zlib) [ 7556.262182] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7556.349735] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7557.935186] Key type lgssc registered [ 7558.259593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7569.001655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7569.006451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.083923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7570.093819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7570.126472] Lustre: lustre-MDT0000: new disk, initializing [ 7570.144811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.149839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7571.515990] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7575.642074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.682114] Lustre: 229330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7575.713764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7575.715664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7575.718264] Lustre: Skipped 1 previous similar message [ 7575.757659] Lustre: lustre-MDT0001: new disk, initializing [ 7575.791424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.805502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7575.808970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7577.095081] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7580.628747] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7582.425036] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7584.641345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.731973] Lustre: lustre-OST0000: new disk, initializing [ 7584.733869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7584.736363] Lustre: 231297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7584.763154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.589729] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7590.230694] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7591.414779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7591.417968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7591.450154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7591.576164] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7591.649327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7593.018335] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7593.099402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7596.512747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7596.515631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.516629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7596.518753] Lustre: Skipped 1 previous similar message [ 7599.819102] Lustre: server umount lustre-OST0000 complete [ 7601.632423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.632739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7601.633135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.633139] Lustre: Skipped 1 previous similar message [ 7601.637282] Lustre: Skipped 1 previous similar message [ 7601.640168] LustreError: Skipped 1 previous similar message [ 7606.752330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7606.755046] Lustre: Skipped 2 previous similar messages [ 7607.126250] Lustre: server umount lustre-MDT0000 complete [ 7608.405488] LustreError: 230128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525635 with bad export cookie 4291679034959908503 [ 7608.409457] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7608.410356] LustreError: 230128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7608.517882] Lustre: server umount lustre-MDT0001 complete [ 7614.197638] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7617.497913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.628241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.874587] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7621.593837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.698121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7622.970546] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7623.903192] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7626.054807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.144818] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7631.016687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7631.079785] Lustre: lustre-OST0001: new disk, initializing [ 7631.082053] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7631.085717] Lustre: 236533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7631.118570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.122489] Lustre: Skipped 1 previous similar message [ 7633.090320] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7636.993805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7636.996728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7637.047584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7638.584133] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7645.261651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7649.044637] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:27:55 (1789525675) [ 7652.320853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.324087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.325016] Lustre: Skipped 2 previous similar messages [ 7652.326613] Lustre: Skipped 1 previous similar message [ 7652.328052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.891952] Lustre: server umount lustre-MDT0000 complete [ 7657.110178] LustreError: 234209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525684 with bad export cookie 4291679034959909385 [ 7657.113060] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7657.114266] LustreError: 234209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7657.230878] Lustre: server umount lustre-MDT0001 complete [ 7667.653461] Lustre: server umount lustre-OST0000 complete [ 7679.196623] Lustre: server umount lustre-OST0001 complete [ 7681.355454] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7684.357561] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7695.619535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.716282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7695.729309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7695.765772] Lustre: lustre-MDT0000: new disk, initializing [ 7695.786908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.793437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7697.131127] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7701.200857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.245242] Lustre: 240470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7701.343752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7702.673608] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7706.402101] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7708.275260] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7710.580508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.700162] Lustre: lustre-OST0000: new disk, initializing [ 7710.701931] Lustre: Skipped 1 previous similar message [ 7710.703979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7710.706411] Lustre: Skipped 2 previous similar messages [ 7710.709073] Lustre: 242442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7712.248414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7712.251031] Lustre: Skipped 1 previous similar message [ 7712.253277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7712.289354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7712.727682] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7716.552411] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7718.067176] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7718.156817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7719.572205] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7719.648711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7722.465440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7722.465804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.467416] LustreError: Skipped 1 previous similar message [ 7722.470821] Lustre: Skipped 1 previous similar message [ 7722.472756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7722.475064] Lustre: Skipped 2 previous similar messages [ 7726.424746] Lustre: server umount lustre-OST0000 complete [ 7732.191720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7732.194336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.844686] Lustre: server umount lustre-MDT0000 complete [ 7735.046051] LustreError: 241268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525762 with bad export cookie 4291679034959911478 [ 7735.048105] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7735.050160] LustreError: 241268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7735.166253] Lustre: server umount lustre-MDT0001 complete [ 7738.043443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.183616] LustreError: 244653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.190943] LustreError: 244653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7738.212250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.214329] Lustre: Skipped 2 previous similar messages [ 7739.556114] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7742.336915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.838535] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7746.380431] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7748.240352] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7750.635644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7752.873468] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7755.419780] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7897.570165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.570169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7897.570975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.573709] Lustre: Skipped 2 previous similar messages [ 7897.576359] Lustre: Skipped 2 previous similar messages [ 7903.371976] Lustre: server umount lustre-OST0000 complete [ 7910.678163] Lustre: server umount lustre-MDT0000 complete [ 7911.904462] LustreError: 247475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525939 with bad export cookie 4291679034959912360 [ 7911.907580] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7911.908063] LustreError: 247475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7912.026419] Lustre: server umount lustre-MDT0001 complete [ 8019.018835] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8019.917946] Key type lgssc unregistered [ 8020.039345] LNet: 248258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8020.042743] LNetError: 248258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8020.055282] LNet: Removed LNI 192.168.202.157@tcp [ 8020.352097] Key type .llcrypt unregistered [ 8020.353523] Key type ._llcrypt unregistered [ 8027.948266] Key type ._llcrypt registered [ 8027.949345] Key type .llcrypt registered [ 8027.984909] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8034.172376] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8034.526373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8034.569891] alg: No test for adler32 (adler32-zlib) [ 8035.415122] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8035.506266] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8037.087174] Key type lgssc registered [ 8037.445576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8051.321888] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8054.555618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.561727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.653870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.665631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8055.703614] Lustre: lustre-MDT0000: new disk, initializing [ 8055.724462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.730318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8057.060121] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8060.956991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8060.994207] Lustre: 252294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8061.014289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8061.016074] Lustre: Skipped 1 previous similar message [ 8061.049704] Lustre: lustre-MDT0001: new disk, initializing [ 8061.077363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8061.088203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8061.091353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8062.343688] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8064.550370] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8066.551983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8066.647440] Lustre: lustre-OST0000: new disk, initializing [ 8066.648863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8066.651624] Lustre: 253926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8066.683754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.511820] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8072.759578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.821916] Lustre: lustre-OST0001: new disk, initializing [ 8072.823477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8072.826066] Lustre: 254787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8072.861433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8073.307246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8073.311177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8073.363852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8073.369545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8074.934196] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8080.554681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8086.288628] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8092.127934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8092.132275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.137826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.179254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.180655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.184839] Lustre: Skipped 1 previous similar message [ 8094.186701] Lustre: Skipped 1 previous similar message [ 8095.465171] Lustre: server umount lustre-MDT0000 complete [ 8096.763252] LustreError: 252288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526124 with bad export cookie 12768963678965862718 [ 8096.766701] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8096.768325] LustreError: 252288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8096.893356] Lustre: server umount lustre-MDT0001 complete [ 8108.482188] Lustre: server umount lustre-OST0000 complete [ 8120.002401] Lustre: server umount lustre-OST0001 complete [ 8127.117327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8133.585143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8139.583661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8146.504586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8160.857861] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8167.630695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.648264] Lustre: lustre-MDT0000: reset Object Index mappings [ 8168.003263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.122575] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8173.026213] LustreError: 259800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8173.038267] LustreError: 259800:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8177.459679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.475462] Lustre: lustre-MDT0001: reset Object Index mappings [ 8177.733416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.983170] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8181.389020] Lustre: 260942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8185.036435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8188.325308] LustreError: 261298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.663990] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8193.520448] LustreError: 261296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.357703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.568141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8194.586930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8194.589207] Lustre: Skipped 1 previous similar message [ 8198.800162] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8199.671973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8204.352487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8206.911800] Lustre: 262811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8209.894912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.897482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.900197] Lustre: Skipped 2 previous similar messages [ 8209.912792] Lustre: Skipped 4 previous similar messages [ 8215.022040] LustreError: 262120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.030812] LustreError: 262120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8215.086859] Lustre: server umount lustre-MDT0000 complete [ 8216.863849] LustreError: 259781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526244 with bad export cookie 12768963678965869396 [ 8216.871252] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.872279] LustreError: 259781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8217.056487] Lustre: server umount lustre-MDT0001 complete [ 8229.409402] Lustre: server umount lustre-OST0000 complete [ 8240.842894] Lustre: server umount lustre-OST0001 complete [ 8250.267813] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:37:56 (1789526276) [ 8257.882718] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8262.619612] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8279.787562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.941685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8279.969603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8280.018607] Lustre: lustre-MDT0000: new disk, initializing [ 8280.076100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.090743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8280.096822] Lustre: Skipped 1 previous similar message [ 8282.137885] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8288.314453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.379888] Lustre: 267058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8288.410854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8288.413419] Lustre: Skipped 1 previous similar message [ 8288.456749] Lustre: lustre-MDT0001: new disk, initializing [ 8288.538140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8288.547859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8288.556787] Lustre: Skipped 1 previous similar message [ 8290.820586] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8296.717629] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8299.835123] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8303.627438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.783651] Lustre: 269025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8304.919757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8304.924118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8305.004930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8307.228917] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8312.253718] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8314.628307] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8314.755216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8316.977259] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8317.100553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8320.483396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.494034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.502231] Lustre: Skipped 2 previous similar messages [ 8320.509566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.512263] Lustre: Skipped 1 previous similar message [ 8324.343989] Lustre: server umount lustre-OST0000 complete [ 8329.696459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8329.700966] LustreError: Skipped 1 previous similar message [ 8329.704212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.709767] Lustre: Skipped 1 previous similar message [ 8329.712619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8332.740421] Lustre: server umount lustre-MDT0000 complete [ 8334.659287] LustreError: 267050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526362 with bad export cookie 12768963678965871482 [ 8334.660220] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.665312] LustreError: 267050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8334.878568] Lustre: server umount lustre-MDT0001 complete [ 8344.512700] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8350.445601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8350.654824] LustreError: 271957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.687651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.690033] Lustre: Skipped 2 previous similar messages [ 8352.685223] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8357.326635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.753703] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8361.297563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8364.962182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8368.440840] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8373.204277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8373.300450] Lustre: lustre-OST0001: new disk, initializing [ 8373.302308] Lustre: Skipped 1 previous similar message [ 8373.304346] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8373.306338] Lustre: Skipped 1 previous similar message [ 8373.309703] Lustre: 274266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8374.677786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8374.681545] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8374.750342] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8376.110386] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8381.688178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8383.747685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8386.674879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8386.781561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8393.695920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8393.699066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.703275] Lustre: Skipped 1 previous similar message [ 8393.705436] LustreError: 273750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.715021] LustreError: 273750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8393.749610] Lustre: server umount lustre-MDT0000 complete [ 8395.755457] LustreError: 274265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526423 with bad export cookie 12768963678965872364 [ 8395.761799] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8395.761960] LustreError: 274265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8395.907631] Lustre: server umount lustre-MDT0001 complete [ 8412.640253] Lustre: 249808:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789526423/real 1789526423] req@ffff8f4319792d80 x1876454080169472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789526439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8412.660481] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.711192] Lustre: 249806:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789526427/real 1789526427] req@ffff8f430632ed80 x1876454080169984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789526443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8415.719866] Lustre: 249806:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8417.891610] Lustre: server umount lustre-OST0000 complete [ 8419.845756] Lustre: server umount lustre-OST0001 complete [ 8429.044537] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8434.603863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.828319] LustreError: 277603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.964496] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8441.657150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.067949] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8445.640375] Lustre: 278745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8449.118992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8452.252947] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8456.457036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.588101] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8459.693329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8459.706810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8464.273117] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8466.159804] Lustre: 280620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8468.523493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8468.562801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8474.870847] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:41:40 (1789526500) [ 8487.905123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8487.911667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.918724] Lustre: Skipped 1 previous similar message [ 8487.922225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.926561] Lustre: Skipped 2 previous similar messages [ 8493.268836] Lustre: server umount lustre-MDT0000 complete [ 8495.354841] LustreError: 280624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526522 with bad export cookie 12768963678965878727 [ 8495.366256] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.584186] LustreError: 277598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.594568] LustreError: 277598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8501.666126] Lustre: server umount lustre-MDT0001 complete [ 8518.111136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8518.175317] Lustre: server umount lustre-OST0000 complete [ 8526.276363] Lustre: server umount lustre-OST0001 complete [ 8529.953639] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8535.496397] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8560.641209] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8565.888304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.983988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8565.996048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8566.032442] Lustre: lustre-MDT0000: new disk, initializing [ 8566.057830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8566.060427] Lustre: Skipped 7 previous similar messages [ 8566.069195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8568.406877] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8574.728559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.800900] Lustre: 285142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8574.957977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8577.269902] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8584.186636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.351812] Lustre: 286777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8587.608819] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8589.895444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8594.430631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8594.550926] Lustre: 287644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8596.312668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8596.319570] Lustre: Skipped 1 previous similar message [ 8596.401638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8597.691297] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8603.452525] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8605.566370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8605.572626] Lustre: Skipped 1 previous similar message [ 8612.817324] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:43:58 (1789526638) [ 8615.911755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8615.916566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.923928] Lustre: Skipped 5 previous similar messages [ 8615.927632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.932047] Lustre: Skipped 8 previous similar messages [ 8620.221426] Lustre: server umount lustre-MDT0000 complete [ 8622.052617] LustreError: 286108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8622.062281] LustreError: 286108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8622.422322] LustreError: 285136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526649 with bad export cookie 12768963678965886679 [ 8622.432306] LustreError: 285136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8622.442953] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8622.659143] Lustre: server umount lustre-MDT0001 complete [ 8635.417066] Lustre: server umount lustre-OST0000 complete [ 8647.419656] Lustre: server umount lustre-OST0001 complete [ 8651.189533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.162397] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8656.778895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.864715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8656.867668] Lustre: Skipped 4 previous similar messages [ 8658.662968] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8662.142465] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8664.841539] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8668.410486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8670.982191] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8673.995986] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.960698] loop1: detected capacity change from 0 to 204800000 [ 8690.119523] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8690.124685] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8690.129278] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8692.243208] loop1: detected capacity change from 0 to 204800000 [ 8694.467603] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.506215] LustreError: 289877:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs19166: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8694.512458] LustreError: 289877:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs19166 [ 8694.514427] LustreError: 289877:0:(mgs_handler.c:517:mgs_target_reg()) fs19166-MDT0001: Error checking target index: rc = -13 [ 8694.517813] LustreError: fs19166-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8694.522280] LustreError: 293631:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 8694.525895] LustreError: 293631:0:(tgt_mount.c:2128:server_put_super()) no obd fs19166-MDT0001 [ 8694.528923] LustreError: 293631:0:(tgt_mount.c:116:server_deregister_mount()) fs19166-MDT0001 not registered [ 8694.543489] Lustre: server umount fs19166-MDT0001 complete [ 8694.545980] LustreError: 293631:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8697.956448] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8697.999694] Lustre: Setting parameter fs19166-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19166-MDT0001 [ 8698.025229] Lustre: srv-fs19166-MDT0001: No data found on store. Initialize space. [ 8698.027744] Lustre: Skipped 4 previous similar messages [ 8698.066918] Lustre: fs19166-MDT0001: new disk, initializing [ 8698.068466] Lustre: Skipped 3 previous similar messages [ 8698.106604] Lustre: srv-fs19166-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8698.109381] Lustre: 294095:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19166-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8698.112596] LustreError: 294095:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19166-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8699.799307] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8703.100677] Lustre: server umount fs19166-MDT0001 complete [ 8708.130219] loop1: detected capacity change from 0 to 204800000 [ 8710.234891] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.361730] Lustre: 295661:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19166-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8710.364486] LustreError: 295661:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19166-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8711.787381] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8713.483035] Lustre: server umount fs19166-MDT0001 complete [ 8722.913499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8722.916332] Lustre: Skipped 6 previous similar messages [ 8723.436895] Lustre: server umount lustre-MDT0000 complete [ 8727.080847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.159442] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8728.811974] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8732.033679] loop1: detected capacity change from 0 to 204800000 [ 8732.652472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8734.173091] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.307455] Lustre: 298102:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19166-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8734.310305] LustreError: 298102:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19166-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8735.849990] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8737.734638] Lustre: server umount fs19166-MDT0001 complete [ 8748.000701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8748.002736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.006511] Lustre: Skipped 6 previous similar messages [ 8751.709495] Lustre: server umount lustre-OST0000 complete [ 8759.327148] Lustre: server umount lustre-MDT0000 complete [ 8760.807370] LustreError: 289869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526788 with bad export cookie 12768963678965889360 [ 8760.811833] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8760.812760] LustreError: 289869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8760.957898] Lustre: server umount lustre-MDT0001 complete [ 8765.125023] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8766.262693] Key type lgssc unregistered [ 8766.395358] LNet: 300443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8766.397749] LNetError: 300443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8766.405301] LNet: Removed LNI 192.168.202.157@tcp [ 8766.764121] Key type .llcrypt unregistered [ 8766.765519] Key type ._llcrypt unregistered [ 8775.562505] Key type ._llcrypt registered [ 8775.564076] Key type .llcrypt registered [ 8775.620514] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8775.954354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.998105] alg: No test for adler32 (adler32-zlib) [ 8776.874871] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8776.971883] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8778.567186] Key type lgssc registered [ 8779.046138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.358369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8782.364018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.557574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.192747] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8788.594314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.797502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8790.504242] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8793.557140] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.784997] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8798.546843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.715666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.717046] LustreError: 303517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8799.726381] LustreError: 303517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8801.262693] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8804.395418] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8804.833910] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.907067] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.914149] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8813.025975] LustreError: 303663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.032551] LustreError: 303663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8816.834783] loop1: detected capacity change from 0 to 204800000 [ 8816.909068] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8816.914396] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8816.919909] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8818.148631] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.155218] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8819.381248] loop1: detected capacity change from 0 to 204800000 [ 8821.961499] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8822.025735] Lustre: fs17838-OST0000: new disk, initializing [ 8822.027709] Lustre: srv-fs17838-OST0000: No data found on store. Initialize space. [ 8822.030638] Lustre: 305421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8822.067910] Lustre: fs17838-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.466285] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8828.388575] LustreError: 303663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.394229] LustreError: 303663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8828.866983] Lustre: server umount fs17838-OST0000 complete [ 8833.948200] loop1: detected capacity change from 0 to 204800000 [ 8836.259933] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8836.341101] Lustre: fs17838-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.679944] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8842.449439] loop2: detected capacity change from 0 to 204800000 [ 8842.537401] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8842.542504] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8842.547912] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8844.574370] loop2: detected capacity change from 0 to 204800000 [ 8845.659204] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.666990] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8846.830956] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8846.865303] LustreError: fs17838-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8846.872114] LustreError: fs17838-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8846.875810] LustreError: 308456:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 8846.879414] LustreError: 308456:0:(tgt_mount.c:2128:server_put_super()) no obd fs17838-OST0001 [ 8846.883236] LustreError: 308456:0:(tgt_mount.c:116:server_deregister_mount()) fs17838-OST0001 not registered [ 8846.895416] Lustre: server umount fs17838-OST0001 complete [ 8846.897589] LustreError: 308456:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8851.114841] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8851.172116] Lustre: fs17838-OST0001: new disk, initializing [ 8851.174363] Lustre: srv-fs17838-OST0001: No data found on store. Initialize space. [ 8851.177688] Lustre: 309009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8851.209222] Lustre: fs17838-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8853.382445] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8857.433605] Lustre: server umount fs17838-OST0000 complete [ 8861.481591] Lustre: server umount fs17838-OST0001 complete [ 8866.451437] loop1: detected capacity change from 0 to 204800000 [ 8868.730970] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.801637] Lustre: fs17838-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.245635] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8874.133166] loop2: detected capacity change from 0 to 204800000 [ 8876.915940] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8879.457308] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8879.587330] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8879.595779] LustreError: 303516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 8883.092565] Lustre: server umount fs17838-OST0000 complete [ 8886.569690] Lustre: server umount fs17838-OST0001 complete [ 8889.830431] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8891.880958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.885930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.895519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8894.944122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.947313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8894.947898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.293918] Lustre: server umount lustre-OST0000 complete [ 8900.064729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.066439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.070768] Lustre: Skipped 1 previous similar message [ 8900.073146] Lustre: Skipped 1 previous similar message [ 8905.184625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8905.189987] Lustre: Skipped 2 previous similar messages [ 8910.304568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8910.308423] Lustre: Skipped 1 previous similar message [ 8912.863119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8912.933191] Lustre: server umount lustre-MDT0000 complete [ 8914.356396] LustreError: 301668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789526941 with bad export cookie 1160768738225463215 [ 8914.359712] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8914.361439] LustreError: 301668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8914.494685] Lustre: server umount lustre-MDT0001 complete [ 8918.917908] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8920.034455] Key type lgssc unregistered [ 8920.175698] LNet: 314917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8920.179303] LNetError: 314917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8920.192475] LNet: Removed LNI 192.168.202.157@tcp [ 8920.565111] Key type .llcrypt unregistered [ 8920.566653] Key type ._llcrypt unregistered [ 8933.496456] Key type ._llcrypt registered [ 8933.497949] Key type .llcrypt registered [ 8933.541249] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8940.912428] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8941.266830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8941.312513] alg: No test for adler32 (adler32-zlib) [ 8942.197698] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 8942.313650] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8943.911225] Key type lgssc registered [ 8944.345147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8959.798188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8959.810225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.900873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8960.920441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8960.963763] Lustre: lustre-MDT0000: new disk, initializing [ 8960.995480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8961.003172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8962.671151] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8967.749903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.800568] Lustre: 318745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8967.827111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8967.829696] Lustre: Skipped 1 previous similar message [ 8967.868146] Lustre: lustre-MDT0001: new disk, initializing [ 8967.908871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.921789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8967.925763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8969.642990] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8974.240650] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8976.426135] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8979.179728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8979.281258] Lustre: lustre-OST0000: new disk, initializing [ 8979.283448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8979.286551] Lustre: 320714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8979.319711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.525314] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8982.140314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8982.144395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8982.186075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8985.965349] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8986.803920] Lustre: Mounted lustre-client - version 2.17.58_39_g97904ba [ 8987.191134] Lustre: Unmounted lustre-client [ 8992.226818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.228346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8992.231132] Lustre: Skipped 1 previous similar message [ 8992.235165] Lustre: Skipped 1 previous similar message [ 8996.723537] Lustre: server umount lustre-OST0000 complete [ 8998.883534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8998.887803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.895662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.470123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.470143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.490358] Lustre: Skipped 1 previous similar message [ 9004.822665] Lustre: server umount lustre-MDT0000 complete [ 9007.287869] LustreError: 318736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527034 with bad export cookie 3480155434856545850 [ 9007.290043] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9007.296582] LustreError: 318736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9007.550646] Lustre: server umount lustre-MDT0001 complete [ 9018.868353] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9022.271149] Key type lgssc unregistered [ 9047.775834] LNet: 1 local NIs in recovery (showing 1): 192.168.202.157@tcp [ 9048.224578] LNet: 322876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9048.243243] LNetError: 322876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9118.954257] LNet: Removed LNI 192.168.202.157@tcp [ 9119.434939] Key type .llcrypt unregistered [ 9119.439966] Key type ._llcrypt unregistered [ 9131.384506] Key type ._llcrypt registered [ 9131.385632] Key type .llcrypt registered [ 9131.445075] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9140.874171] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9141.565790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9141.574420] alg: No test for adler32 (adler32-zlib) [ 9142.534979] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9142.656295] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 9144.279869] Key type lgssc registered [ 9144.948297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9163.257845] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:53:09 (1789527189) [ 9167.541851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9167.553846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.665621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9168.686151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9168.732063] Lustre: lustre-MDT0000: new disk, initializing [ 9168.783849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.792502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9170.857396] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9178.555955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.662702] Lustre: 326067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.733547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9178.738959] Lustre: Skipped 1 previous similar message [ 9178.791210] Lustre: lustre-MDT0001: new disk, initializing [ 9178.879212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9178.902080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9178.908290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9182.440869] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9189.610260] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9193.482688] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9197.929538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.151755] Lustre: lustre-OST0000: new disk, initializing [ 9198.154897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9198.162820] Lustre: 328035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9198.225051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9202.384258] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9208.355129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9208.364513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9208.434154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9208.544335] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9218.549072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9218.549523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.564256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9218.571955] Lustre: Skipped 1 previous similar message [ 9218.583410] Lustre: Skipped 1 previous similar message [ 9223.654457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9223.659323] Lustre: Skipped 1 previous similar message [ 9223.831646] Lustre: server umount lustre-OST0000 complete [ 9228.769703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.773079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.783265] Lustre: Skipped 1 previous similar message [ 9233.071107] Lustre: server umount lustre-MDT0000 complete [ 9235.639471] LustreError: 328045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527263 with bad export cookie 179455906610622910 [ 9235.641694] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9235.648719] LustreError: 328045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9235.898619] Lustre: server umount lustre-MDT0001 complete [ 9242.649121] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9244.469563] Key type lgssc unregistered [ 9244.704525] LNet: 329941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9244.712815] LNetError: 329941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9244.722973] LNet: Removed LNI 192.168.202.157@tcp [ 9245.336786] Key type .llcrypt unregistered [ 9245.338801] Key type ._llcrypt unregistered [ 9260.045864] Key type ._llcrypt registered [ 9260.050590] Key type .llcrypt registered [ 9260.135381] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9260.902635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9260.947516] alg: No test for adler32 (adler32-zlib) [ 9261.956872] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9262.163933] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 9263.807497] Key type lgssc registered [ 9264.573656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9270.169920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9270.184711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.519050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9275.181979] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9280.996259] LustreError: 331190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.009235] LustreError: 331190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9281.976467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.236595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9285.635852] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9291.150555] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9295.455280] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9299.935382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9300.128864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.875927] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9308.753935] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9324.922899] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9325.928771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9328.608517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9328.619431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.631839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9329.516986] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 9332.195082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.198711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.206035] Lustre: Skipped 1 previous similar message [ 9334.618916] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 9334.626477] Lustre: Skipped 2 previous similar messages [ 9334.855247] Lustre: server umount lustre-MDT0000 complete [ 9337.312600] LustreError: 331189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.320167] LustreError: 331189:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9337.511087] LustreError: 331171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527364 with bad export cookie 10780257545707989507 [ 9337.518254] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9339.741793] LustreError: 331186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.760573] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.57@tcp (stopping) [ 9342.431904] LustreError: 333487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.432245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.760804] Lustre: server umount lustre-MDT0001 complete [ 9352.369139] Lustre: server umount lustre-OST0000 complete [ 9358.228756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.018859] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9365.131537] Lustre: 335396:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.157@tcp [ 9367.984592] Lustre: server umount MGS complete [ 9379.589506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.067292] LustreError: 335963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.378606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.788450] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9389.440887] LustreError: 335964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.482407] LustreError: 335964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9400.848088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.510295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9407.861369] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9420.946927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9421.445421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.986372] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9435.577415] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9437.484722] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9443.344780] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9444.617807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9450.989247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9450.994729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9451.005533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9451.009435] Lustre: Skipped 1 previous similar message [ 9451.572988] Lustre: server umount lustre-OST0000 complete [ 9457.647737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9457.658729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9457.670660] Lustre: Skipped 1 previous similar message [ 9461.827484] Lustre: server umount lustre-MDT0000 complete [ 9462.756211] LustreError: 335968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.785244] LustreError: 335968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9466.076867] LustreError: 335948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527493 with bad export cookie 10780257545707990347 [ 9466.082744] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9466.087726] LustreError: 335948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9476.769438] Lustre: server umount lustre-MDT0001 complete [ 9488.811863] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9491.347369] Key type lgssc unregistered [ 9491.615518] LNet: 339405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9491.625567] LNetError: 339405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9492.650763] LNet: Removed LNI 192.168.202.157@tcp [ 9493.460364] Key type .llcrypt unregistered [ 9493.462820] Key type ._llcrypt unregistered [ 9513.116403] Key type ._llcrypt registered [ 9513.120506] Key type .llcrypt registered [ 9513.236586] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9530.963571] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9532.016291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9532.043188] alg: No test for adler32 (adler32-zlib) [ 9533.099703] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 9533.306995] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 9534.999222] Key type lgssc registered [ 9535.837309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9569.358583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9569.379261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9570.753224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9570.810282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9570.876414] Lustre: lustre-MDT0000: new disk, initializing [ 9570.966273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.987138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9574.837409] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9584.573308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9584.729527] Lustre: 342566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9584.772218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9584.776650] Lustre: Skipped 1 previous similar message [ 9584.854392] Lustre: lustre-MDT0001: new disk, initializing [ 9584.948598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9584.989337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9585.021979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9588.657547] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9595.554857] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9600.843342] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9606.899349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9607.092439] Lustre: lustre-OST0000: new disk, initializing [ 9607.095309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9607.100398] Lustre: 344536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9607.180193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9612.940023] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9615.396340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9615.404874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9615.493846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9620.490548] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9623.802271] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9623.976523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9627.944553] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9628.119621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9630.695551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9630.696138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9630.704511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9630.715988] Lustre: Skipped 1 previous similar message [ 9630.728020] Lustre: Skipped 1 previous similar message [ 9635.952941] Lustre: server umount lustre-OST0000 complete [ 9640.932982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.936682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9640.953712] Lustre: Skipped 1 previous similar message [ 9640.965100] Lustre: Skipped 1 previous similar message [ 9645.889044] Lustre: server umount lustre-MDT0000 complete [ 9646.063111] LustreError: 342578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9646.081583] LustreError: 342578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9648.986317] LustreError: 342560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527676 with bad export cookie 14968007643876173500 [ 9648.991318] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9648.991461] LustreError: 342560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9649.008666] LustreError: 340787:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 9649.297918] Lustre: server umount lustre-MDT0001 complete [ 9657.178340] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:01:22 (1789527682) [ 9672.272220] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9681.324903] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9709.887621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.065105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9710.081876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9710.141282] Lustre: lustre-MDT0000: new disk, initializing [ 9710.191288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.201984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9713.667694] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9722.887328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.977383] Lustre: 349303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9723.038071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9723.040319] Lustre: Skipped 1 previous similar message [ 9723.149783] Lustre: lustre-MDT0001: new disk, initializing [ 9723.307779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9723.337667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9723.356525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9727.464685] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9737.020834] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9741.653978] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9748.113147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.390520] Lustre: lustre-OST0000: new disk, initializing [ 9748.395666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9748.405167] Lustre: 351271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9748.484106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.413413] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9757.268822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9757.283223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9757.364293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9763.570828] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9798.630304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9798.633203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.649943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9798.655734] Lustre: Skipped 1 previous similar message [ 9800.583692] Lustre: server umount lustre-OST0000 complete [ 9805.280506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9805.287971] LustreError: Skipped 1 previous similar message [ 9805.291482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.301353] Lustre: Skipped 1 previous similar message [ 9805.316173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9810.831117] Lustre: server umount lustre-MDT0000 complete [ 9813.984824] LustreError: 349310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9814.006050] LustreError: 349310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9814.520891] LustreError: 349296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789527841 with bad export cookie 14968007643876174382 [ 9814.532644] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.901263] Lustre: server umount lustre-MDT0001 complete [ 9826.361551] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9828.673864] Key type lgssc unregistered [ 9829.002873] LNet: 353435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9829.014396] LNetError: 353435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9829.034757] LNet: Removed LNI 192.168.202.157@tcp [ 9829.905137] Key type .llcrypt unregistered [ 9829.907683] Key type ._llcrypt unregistered [ 9910.705062] Key type ._llcrypt registered [ 9910.708914] Key type .llcrypt registered [ 9910.828378] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9911.043528] Key type .llcrypt unregistered [ 9911.046513] Key type ._llcrypt unregistered [ 9969.210165] Key type ._llcrypt registered [ 9969.213696] Key type .llcrypt registered [ 9969.306301] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9969.480153] Key type .llcrypt unregistered [ 9969.483628] Key type ._llcrypt unregistered [ 9989.675648] Key type ._llcrypt registered [ 9989.678597] Key type .llcrypt registered [ 9989.801889] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9989.967039] Key type .llcrypt unregistered [ 9989.969892] Key type ._llcrypt unregistered [10008.707581] Key type ._llcrypt registered [10008.708950] Key type .llcrypt registered [10008.749815] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10008.892705] Key type .llcrypt unregistered [10008.893987] Key type ._llcrypt unregistered [10050.448336] Key type ._llcrypt registered [10050.450876] Key type .llcrypt registered [10050.526726] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10050.700188] Key type .llcrypt unregistered [10050.702958] Key type ._llcrypt unregistered [10068.238389] Key type ._llcrypt registered [10068.240094] Key type .llcrypt registered [10068.308804] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10068.478514] Key type .llcrypt unregistered [10068.480978] Key type ._llcrypt unregistered [10085.927578] Key type ._llcrypt registered [10085.928805] Key type .llcrypt registered [10085.988245] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10086.161237] Key type .llcrypt unregistered [10086.162855] Key type ._llcrypt unregistered [10105.223985] Key type ._llcrypt registered [10105.226464] Key type .llcrypt registered [10105.305485] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10105.503309] Key type .llcrypt unregistered [10105.508345] Key type ._llcrypt unregistered [10125.741779] Key type ._llcrypt registered [10125.743709] Key type .llcrypt registered [10125.800812] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10125.979513] Key type .llcrypt unregistered [10125.980796] Key type ._llcrypt unregistered [10146.103381] Key type ._llcrypt registered [10146.108090] Key type .llcrypt registered [10146.195705] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10146.445583] Key type .llcrypt unregistered [10146.447606] Key type ._llcrypt unregistered [10186.071356] Key type ._llcrypt registered [10186.073010] Key type .llcrypt registered [10186.167134] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10186.318364] Key type .llcrypt unregistered [10186.319971] Key type ._llcrypt unregistered [10203.673984] Key type ._llcrypt registered [10203.675652] Key type .llcrypt registered [10203.767951] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10203.941638] Key type .llcrypt unregistered [10203.943277] Key type ._llcrypt unregistered [10219.024522] Key type ._llcrypt registered [10219.026499] Key type .llcrypt registered [10219.082821] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10219.242444] Key type .llcrypt unregistered [10219.244556] Key type ._llcrypt unregistered [10247.909362] Key type ._llcrypt registered [10247.910832] Key type .llcrypt registered [10247.964703] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10256.654676] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10257.044920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10257.107587] alg: No test for adler32 (adler32-zlib) [10257.982753] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [10258.097984] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [10259.711200] Key type lgssc registered [10260.231527] Lustre: Echo OBD driver; http://www.lustre.org/ [10276.478625] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10276.487841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.581505] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10277.610353] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10277.651355] Lustre: tst250fs-MDT0000: new disk, initializing [10277.682482] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.691834] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10279.280961] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10284.258862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.308759] Lustre: 373357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10284.340531] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10284.343142] Lustre: Skipped 1 previous similar message [10284.381309] Lustre: tst250fs-MDT0001: new disk, initializing [10284.422117] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10284.439404] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10284.444604] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10286.088994] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10289.148474] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10292.222749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10292.353251] Lustre: tst250fs-OST0000: new disk, initializing [10292.356954] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10292.360931] Lustre: 374988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10292.413041] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10294.726233] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10294.731265] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10294.781252] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10295.107612] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10300.571343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10300.654405] Lustre: tst250fs-OST0001: new disk, initializing [10300.657118] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10300.660975] Lustre: 375859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10300.704959] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10301.397116] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10301.400723] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10301.456183] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10303.258242] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10306.575709] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10320.352205] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10320.359100] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.366645] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10321.890479] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10321.891127] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.013970] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10327.019432] Lustre: Skipped 6 previous similar messages [10332.129304] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10332.133378] Lustre: Skipped 2 previous similar messages [10334.175112] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10334.252040] Lustre: server umount tst250fs-MDT0000 complete [10336.116031] LustreError: 374153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789528363 with bad export cookie 5861751956753231832 [10336.118973] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10336.128756] LustreError: 374153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10336.340628] Lustre: server umount tst250fs-MDT0001 complete [10347.660566] Lustre: server umount tst250fs-OST0000 complete [10360.139530] Lustre: server umount tst250fs-OST0001 complete [10370.224548] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10374.460095] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10389.100762] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:13:35 (1789528415) [10389.890540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10400.178271] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10405.408447] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10420.932124] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10143 sec ========= 23:14:06 (1789528446) [10421.842277] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:14:07 (1789528447) === [10423.516568] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:14:09 (1789528449) === [10444.166824] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10445.831646] Key type lgssc unregistered [10446.032605] LNet: 384972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10446.038455] LNetError: 384972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10446.051952] LNet: Removed LNI 192.168.202.157@tcp [10446.652328] Key type .llcrypt unregistered [10446.654868] Key type ._llcrypt unregistered