[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501486842 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006791] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011060] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022549] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027107] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029123] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031154] *** VALIDATE tmpfs *** [ 0.033481] *** VALIDATE proc *** [ 0.034266] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043645] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.045200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054357] rcu: Hierarchical SRCU implementation. [ 0.056561] smp: Bringing up secondary CPUs ... [ 0.057668] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.065203] smp: Brought up 1 node, 4 CPUs [ 0.067015] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.087025] node 0 deferred pages initialised in 17ms [ 0.089531] devtmpfs: initialized [ 0.090297] x86/mm: Memory block size: 128MB [ 0.092568] gcov: version magic: 0x41383552 [ 0.095238] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.097089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.099326] pinctrl core: initialized pinctrl subsystem [ 0.100127] [ 0.100747] ************************************************************* [ 0.103012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.104008] ** ** [ 0.105008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.107010] ** ** [ 0.108007] ** This means that this kernel is built to expose internal ** [ 0.110016] ** IOMMU data structures, which may compromise security on ** [ 0.113017] ** your system. ** [ 0.114013] ** ** [ 0.116012] ** If you see this message and you are not debugging the ** [ 0.117010] ** kernel, report this immediately to your vendor! ** [ 0.119007] ** ** [ 0.120009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122013] ************************************************************* [ 0.124623] NET: Registered protocol family 16 [ 0.125420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.127039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.129057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.132135] cpuidle: using governor menu [ 0.134599] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.136395] PCI: Using configuration type 1 for base access [ 0.137109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.144058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.145120] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.146130] cryptd: max_cpu_qlen set to 1000 [ 0.147193] ACPI: Added _OSI(Module Device) [ 0.148051] ACPI: Added _OSI(Processor Device) [ 0.149018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.151016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.155747] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.160565] ACPI: Interpreter enabled [ 0.161048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.162009] ACPI: Using IOAPIC for interrupt routing [ 0.164135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.167465] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.174018] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.176038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.179025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.182091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.187361] acpiphp: Slot [2] registered [ 0.188101] acpiphp: Slot [5] registered [ 0.189096] acpiphp: Slot [6] registered [ 0.190077] acpiphp: Slot [7] registered [ 0.191093] acpiphp: Slot [8] registered [ 0.192208] acpiphp: Slot [9] registered [ 0.194111] acpiphp: Slot [10] registered [ 0.195105] acpiphp: Slot [3] registered [ 0.196086] acpiphp: Slot [4] registered [ 0.197085] acpiphp: Slot [11] registered [ 0.198077] acpiphp: Slot [12] registered [ 0.199133] acpiphp: Slot [13] registered [ 0.200082] acpiphp: Slot [14] registered [ 0.201090] acpiphp: Slot [15] registered [ 0.202085] acpiphp: Slot [16] registered [ 0.203076] acpiphp: Slot [17] registered [ 0.204083] acpiphp: Slot [18] registered [ 0.205045] acpiphp: Slot [19] registered [ 0.207067] acpiphp: Slot [20] registered [ 0.208095] acpiphp: Slot [21] registered [ 0.209093] acpiphp: Slot [22] registered [ 0.211129] acpiphp: Slot [23] registered [ 0.212116] acpiphp: Slot [24] registered [ 0.214104] acpiphp: Slot [25] registered [ 0.215087] acpiphp: Slot [26] registered [ 0.216093] acpiphp: Slot [27] registered [ 0.217099] acpiphp: Slot [28] registered [ 0.219083] acpiphp: Slot [29] registered [ 0.220168] acpiphp: Slot [30] registered [ 0.221095] acpiphp: Slot [31] registered [ 0.223085] PCI host bridge to bus 0000:00 [ 0.224021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.226020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.228021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.230022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.233025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.235025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.237189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.240118] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.243418] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.251701] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.258038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.260017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.267639] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.270829] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.276645] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.281015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.293017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.296857] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.302095] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.308017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.338025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.359591] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.367018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.377018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.393019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.404807] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.410015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.419022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.431020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.443945] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.452019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.459017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.479022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.491878] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.502085] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.510016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.525017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.535255] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.543019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.550014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.565021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.579725] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.584497] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.587347] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.589494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.592227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.597148] iommu: Default domain type: Passthrough [ 0.598000] SCSI subsystem initialized [ 0.599129] ACPI: bus type USB registered [ 0.600136] usbcore: registered new interface driver usbfs [ 0.602086] usbcore: registered new interface driver hub [ 0.604098] usbcore: registered new device driver usb [ 0.606180] pps_core: LinuxPPS API ver. 1 registered [ 0.607010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.610090] PTP clock support registered [ 0.612059] EDAC MC: Ver: 3.0.0 [ 0.614108] PCI: Using ACPI for IRQ routing [ 0.615503] NetLabel: Initializing [ 0.617011] NetLabel: domain hash size = 128 [ 0.617947] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619092] NetLabel: unlabeled traffic allowed by default [ 0.621129] vgaarb: loaded [ 0.622268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.624013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.629807] clocksource: Switched to clocksource kvm-clock [ 0.723673] VFS: Disk quotas dquot_6.6.0 [ 0.725297] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727873] *** VALIDATE ramfs *** [ 0.729119] *** VALIDATE hugetlbfs *** [ 0.730668] pnp: PnP ACPI init [ 0.733079] pnp: PnP ACPI: found 6 devices [ 0.751280] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.754624] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.756912] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.758790] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.761258] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.763740] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.766575] NET: Registered protocol family 2 [ 0.769232] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.774199] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.778219] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.784216] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.788259] TCP: Hash tables configured (established 65536 bind 65536) [ 0.791486] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.794932] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.797830] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.800984] NET: Registered protocol family 1 [ 0.804250] RPC: Registered named UNIX socket transport module. [ 0.806494] RPC: Registered udp transport module. [ 0.808161] RPC: Registered tcp transport module. [ 0.809737] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.812256] NET: Registered protocol family 44 [ 0.813874] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.816324] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.818619] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.821033] PCI: CLS 0 bytes, default 64 [ 0.822708] Unpacking initramfs... [ 2.174710] debug: unmapping init [mem 0xffff94397cc54000-0xffff94397ffbffff] [ 2.178962] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.181280] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.183989] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.663640] Initialise system trusted keyrings [ 2.664672] Key type blacklist registered [ 2.666297] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.674649] zbud: loaded [ 2.678082] *** VALIDATE nfs *** [ 2.679374] *** VALIDATE nfs4 *** [ 2.680714] pstore: using deflate compression [ 2.683449] Platform Keyring initialized [ 2.777854] NET: Registered protocol family 38 [ 2.779850] Key type asymmetric registered [ 2.781712] Asymmetric key parser 'x509' registered [ 2.784177] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.787687] io scheduler mq-deadline registered [ 2.788823] io scheduler kyber registered [ 2.789850] io scheduler bfq registered [ 2.791516] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.794603] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.797201] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.800229] ACPI: Power Button [PWRF] [ 2.806296] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.814465] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.830356] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.840156] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.855527] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.881651] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.910979] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.915886] Non-volatile memory driver v1.3 [ 2.917607] Linux agpgart interface v0.103 [ 2.953546] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.957408] vda: detected capacity change from 0 to 76791808 [ 2.977410] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.980674] vdb: detected capacity change from 0 to 1073741824 [ 2.996027] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.999242] vdc: detected capacity change from 0 to 2621440000 [ 3.015589] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.018457] vdd: detected capacity change from 0 to 2621440000 [ 3.034908] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.038420] vde: detected capacity change from 0 to 4294967296 [ 3.055207] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.058173] vdf: detected capacity change from 0 to 4294967296 [ 3.064290] libphy: Fixed MDIO Bus: probed [ 3.072473] usbcore: registered new interface driver usbserial_generic [ 3.075974] usbserial: USB Serial support registered for generic [ 3.078401] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.082889] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.084902] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.087492] mousedev: PS/2 mouse device common for all mice [ 3.090454] rtc_cmos 00:05: RTC can wake from S4 [ 3.093684] rtc_cmos 00:05: registered as rtc0 [ 3.095541] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.098470] intel_pstate: CPU model not supported [ 3.098531] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.105159] hid: raw HID events driver (C) Jiri Kosina [ 3.105588] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.107516] usbcore: registered new interface driver usbhid [ 3.111604] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.112887] usbhid: USB HID core driver [ 3.113663] drop_monitor: Initializing network drop monitor service [ 3.120902] Initializing XFRM netlink socket [ 3.123249] NET: Registered protocol family 10 [ 3.127227] Segment Routing with IPv6 [ 3.129241] NET: Registered protocol family 17 [ 3.132341] mpls_gso: MPLS GSO support [ 3.138616] RAS: Correctable Errors collector initialized. [ 3.140763] AVX version of gcm_enc/dec engaged. [ 3.142438] AES CTR mode by8 optimization enabled [ 3.218414] sched_clock: Marking stable (3218389941, 0)->(4155346178, -936956237) [ 3.222216] registered taskstats version 1 [ 3.224347] Loading compiled-in X.509 certificates [ 3.226508] zswap: loaded using pool lzo/zbud [ 3.252981] Key type big_key registered [ 3.264248] Key type encrypted registered [ 3.265977] ima: No TPM chip found, activating TPM-bypass! [ 3.268000] ima: Allocated hash algorithm: sha1 [ 3.269585] ima: No architecture policies found [ 3.271194] evm: Initialising EVM extended attributes: [ 3.272906] evm: security.selinux [ 3.274229] evm: security.ima [ 3.275278] evm: security.capability [ 3.276570] evm: HMAC attrs: 0x1 [ 3.279093] rtc_cmos 00:05: setting system clock to 2026-09-09 16:05:57 UTC (1788969957) [ 3.285616] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.288964] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 3.297076] Write protecting the kernel read-only data: 28672k [ 3.300472] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.303349] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 3.336535] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.345140] systemd[1]: Detected virtualization kvm. [ 3.346492] systemd[1]: Detected architecture x86-64. [ 3.348667] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.374067] systemd[1]: No hostname configured. [ 3.376091] systemd[1]: Set hostname to . [ 3.378397] random: systemd: uninitialized urandom read (16 bytes read) [ 3.380961] systemd[1]: Initializing machine ID from random generator. [ 3.524782] random: systemd: uninitialized urandom read (16 bytes read) [ 3.527637] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.531478] random: systemd: uninitialized urandom read (16 bytes read) [ 3.534764] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.539253] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.202777] device-mapper: uevent: version 1.0.3 [ 4.205112] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.966287] random: fast init done [ 5.003300] virtio_net virtio0 ens2: renamed from eth0 [ 5.116451] scsi host0: ata_piix [ 5.161565] scsi host1: ata_piix [ 5.163364] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.165482] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.807558] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.785103] random: crng init done [ 9.786339] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.292552] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.546715] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.800674] SELinux: Disabled at runtime. [ 11.865471] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.873352] systemd[1]: Detected virtualization kvm. [ 11.875391] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.368590] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.370947] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.374938] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.381125] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.383013] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.390112] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.397970] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.551142] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.876048] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.311317] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.354454] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.519730] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.539600] EDAC sbridge: Ver: 1.1.2 [ 15.216660] Key type dns_resolver registered [ 15.510389] NFS: Registering the id_resolver key type [ 15.512322] Key type id_resolver registered [ 15.513858] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-server login: [ 55.189273] libcfs: loading out-of-tree module taints kernel. [ 55.245109] Key type ._llcrypt registered [ 55.246808] Key type .llcrypt registered [ 55.402675] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 74.257502] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 75.943574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.963727] alg: No test for adler32 (adler32-zlib) [ 77.299412] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 78.326525] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 80.137046] Key type lgssc registered [ 82.457230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.169208] hrtimer: interrupt took 5667141 ns [ 102.489259] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.376576] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 160.340360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.399956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.687639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.747143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.903868] Lustre: lustre-MDT0000: new disk, initializing [ 162.037266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.063058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.161727] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.518724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.630205] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.660232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.663963] Lustre: Skipped 1 previous similar message [ 185.731767] Lustre: lustre-MDT0001: new disk, initializing [ 185.803298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.838034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.878243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.232142] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.054424] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.896247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.295436] Lustre: lustre-OST0000: new disk, initializing [ 210.305432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.317536] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.473958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.003718] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.741494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.753578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.823137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.222749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.399563] Lustre: lustre-OST0001: new disk, initializing [ 237.409651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.420416] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.509716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.724394] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.874758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.898431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.992457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.802859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.718309] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.375852] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.377638] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing yml_node [ 288.399350] Lustre: DEBUG MARKER: Client: 2.17.58.42 [ 291.112715] Lustre: DEBUG MARKER: MDS: 2.17.58.42 [ 294.100435] Lustre: DEBUG MARKER: OSS: 2.17.58.42 [ 295.836541] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 12:10:48 EDT 2026 [ 314.306833] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 316.016846] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.258164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.260054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.275189] Lustre: Skipped 1 previous similar message [ 332.296399] Lustre: Skipped 3 previous similar messages [ 337.376556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.385447] Lustre: Skipped 3 previous similar messages [ 338.118684] Lustre: server umount lustre-MDT0000 complete [ 341.912722] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970296 with bad export cookie 12489036773776276009 [ 341.929030] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.500563] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.523995] Lustre: Skipped 2 previous similar messages [ 342.530697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.533614] Lustre: Skipped 1 previous similar message [ 346.529708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.537572] Lustre: Skipped 1 previous similar message [ 351.714684] LustreError: 10091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.715373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.735493] LustreError: 10091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 351.756874] Lustre: Skipped 1 previous similar message [ 352.743108] LustreError: 6512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.765579] LustreError: 6512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 356.319128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.624554] Lustre: server umount lustre-MDT0001 complete [ 361.375425] Lustre: server umount lustre-OST0000 complete [ 366.040819] Lustre: server umount lustre-OST0001 complete [ 372.626347] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 386.744950] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 421.646044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.138485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.227970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.384658] Lustre: lustre-MDT0000: new disk, initializing [ 422.562506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.593444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.537650] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 440.175112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.320433] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.395131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.405225] Lustre: Skipped 1 previous similar message [ 440.520830] Lustre: lustre-MDT0001: new disk, initializing [ 440.683061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.757803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.816642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.761399] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 455.429214] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.537596] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.060752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.395648] Lustre: lustre-OST0000: new disk, initializing [ 468.399604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 468.407349] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 468.505604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.437728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.446721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.543768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.161445] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 483.674239] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.404846] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.663114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.853707] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.048197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.102809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.109353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.116509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.116520] Lustre: Skipped 2 previous similar messages [ 496.129611] Lustre: Skipped 2 previous similar messages [ 496.159238] LustreError: Skipped 1 previous similar message [ 501.275927] Lustre: server umount lustre-OST0000 complete [ 506.336923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.357127] Lustre: Skipped 1 previous similar message [ 511.801402] Lustre: server umount lustre-MDT0000 complete [ 516.069371] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970470 with bad export cookie 12489036773776277542 [ 516.071387] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.082943] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 516.377565] Lustre: server umount lustre-MDT0001 complete [ 523.527137] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:14:36 (1788970476) [ 544.764284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.311822] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.394818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.862238] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 550.896588] LustreError: 19917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.976946] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.163870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.264988] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 572.638697] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 580.035591] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.355531] Lustre: Failing over lustre-MDT0000 [ 582.626329] Lustre: server umount lustre-MDT0000 complete [ 584.455444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.466934] LustreError: 19912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.502083] Lustre: Skipped 1 previous similar message [ 601.064867] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788970538/real 1788970538] req@ffff9438c302b0c0 x1875871040438784/t0(0) o400->MGC192.168.202.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788970554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 601.102142] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.140846] LustreError: 19912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.163049] LustreError: 19912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 603.972022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.328890] LustreError: 3636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9439d437b840 x1875871040441984/t0(0) o250->MGC192.168.202.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 612.768658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.781330] Lustre: Skipped 1 previous similar message [ 612.856635] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.806691] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 617.954104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.982994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 618.001699] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 629.074379] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 634.337332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.368146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.379995] Lustre: Skipped 5 previous similar messages [ 640.104943] Lustre: server umount lustre-MDT0000 complete [ 643.861177] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970598 with bad export cookie 12489036773776278956 [ 643.872426] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.873604] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 644.392439] Lustre: server umount lustre-MDT0001 complete [ 653.327835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.933541] LustreError: 23254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.952683] LustreError: 23254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 658.373174] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 666.698589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.690166] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 680.357311] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 687.095312] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.346487] Lustre: Failing over lustre-MDT0000 [ 689.519746] Lustre: server umount lustre-MDT0000 complete [ 689.635986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.645369] Lustre: Skipped 1 previous similar message [ 708.063326] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788970646/real 1788970646] req@ffff9438c37f7840 x1875871040470016/t0(0) o400->MGC192.168.202.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788970662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 708.090815] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.432258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.304721] LustreError: 23254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.322806] LustreError: 23254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 719.333891] LustreError: 3636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9439c844c000 x1875871040472704/t0(0) o250->MGC192.168.202.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 719.689126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 724.521724] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 724.965378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 724.975430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 724.981870] Lustre: Skipped 1 previous similar message [ 725.011691] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 733.317357] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 740.326370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 740.334804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.342987] Lustre: Skipped 1 previous similar message [ 740.345665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 740.348274] Lustre: Skipped 2 previous similar messages [ 744.013525] Lustre: server umount lustre-MDT0000 complete [ 749.080201] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970703 with bad export cookie 12489036773776279894 [ 749.106132] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.112694] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 750.006705] Lustre: server umount lustre-MDT0001 complete [ 762.481244] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:18:35 (1788970715) [ 778.888628] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 789.077607] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 835.523859] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 846.435552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.667585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 846.688227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 846.752279] Lustre: lustre-MDT0000: new disk, initializing [ 846.810823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.813920] Lustre: Skipped 3 previous similar messages [ 846.825260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 851.234675] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 861.997904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.157254] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 862.500783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 866.477605] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 875.719599] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.622939] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.661750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.024655] Lustre: 31598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 889.098084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 889.172090] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 889.216864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 889.235192] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 892.456096] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 901.600478] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.414676] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 930.273431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.276541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.280847] LustreError: Skipped 1 previous similar message [ 930.284986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.300347] Lustre: Skipped 2 previous similar messages [ 930.312536] Lustre: Skipped 2 previous similar messages [ 932.961765] Lustre: server umount lustre-OST0000 complete [ 939.488508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 951.263159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 951.500923] Lustre: server umount lustre-MDT0000 complete [ 954.927611] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970909 with bad export cookie 12489036773776280300 [ 954.936204] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.938732] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 955.340627] Lustre: server umount lustre-MDT0001 complete [ 965.614714] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 968.273704] Key type lgssc unregistered [ 968.550503] LNet: 33574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.555402] LNetError: 33574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.577971] LNet: Removed LNI 192.168.202.135@tcp [ 969.437187] Key type .llcrypt unregistered [ 969.438680] Key type ._llcrypt unregistered [ 996.556618] Key type ._llcrypt registered [ 996.559985] Key type .llcrypt registered [ 996.651657] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 998.024413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.132053] alg: No test for adler32 (adler32-zlib) [ 999.292846] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 999.542381] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1001.215330] Key type lgssc registered [ 1002.543327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.316549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.339415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.955126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.407276] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1022.432685] LustreError: 34986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.458879] LustreError: 34986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1024.704712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.059200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.268271] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1036.601626] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1042.895327] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1049.288222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.669175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.752396] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1055.344265] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1063.116645] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.896778] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:23:51 (1788971031) [ 1082.214560] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1084.589821] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1095.143436] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:24:06 (1788971046) [ 1107.957488] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:24:20 (1788971060) [ 1109.569373] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1121.093095] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1121.097816] Lustre: Skipped 2 previous similar messages [ 1123.174609] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:24:36 (1788971076) [ 1126.804952] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1126.809369] Lustre: Skipped 3 previous similar messages [ 1148.499308] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1148.502985] Lustre: Skipped 6 previous similar messages [ 1151.545408] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:25:04 (1788971104) [ 1154.369512] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1157.706995] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1167.139169] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:25:20 (1788971120) [ 1168.977545] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1168.983365] Lustre: Skipped 3 previous similar messages [ 1184.372513] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:25:37 (1788971137) [ 1200.615157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.634060] Lustre: Skipped 1 previous similar message [ 1200.635601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.637774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1200.655196] Lustre: Skipped 1 previous similar message [ 1205.735283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.740045] Lustre: Skipped 1 previous similar message [ 1206.756795] Lustre: server umount lustre-OST0000 complete [ 1214.439326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.451359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.469058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.972860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.747528] Lustre: server umount lustre-MDT0000 complete [ 1221.087767] LustreError: 36031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.105953] LustreError: 36031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1221.705442] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971175 with bad export cookie 16741069186129688270 [ 1221.709149] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.723512] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1222.302272] Lustre: server umount lustre-MDT0001 complete [ 1235.951772] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1239.217025] Key type lgssc unregistered [ 1239.613730] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.620971] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.649144] LNet: Removed LNI 192.168.202.135@tcp [ 1240.878507] Key type .llcrypt unregistered [ 1240.882965] Key type ._llcrypt unregistered [ 1263.038495] Key type ._llcrypt registered [ 1263.043535] Key type .llcrypt registered [ 1263.226534] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 1282.941261] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1284.551518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1284.567935] alg: No test for adler32 (adler32-zlib) [ 1285.583836] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1285.828834] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1287.503222] Key type lgssc registered [ 1288.809294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.166333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.197898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.625497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1330.708597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1330.876697] Lustre: lustre-MDT0000: new disk, initializing [ 1331.099520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.137199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1336.951993] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1350.402598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.608565] Lustre: 46007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1350.692810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1350.696535] Lustre: Skipped 1 previous similar message [ 1350.764654] Lustre: lustre-MDT0001: new disk, initializing [ 1350.855860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.901611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1350.926576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1357.301092] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1367.420087] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.380638] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1380.950549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.294388] Lustre: lustre-OST0000: new disk, initializing [ 1381.297981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1381.302760] Lustre: 47977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1381.380794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.628254] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1389.624258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1389.635492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1389.826782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1399.062594] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1404.887651] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1405.111440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1409.794485] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.038603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1415.652315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1415.654063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.670756] LustreError: Skipped 1 previous similar message [ 1415.672256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1415.693248] Lustre: Skipped 1 previous similar message [ 1415.741499] Lustre: Skipped 1 previous similar message [ 1420.773328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.787532] Lustre: Skipped 1 previous similar message [ 1425.892903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.400365] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1426.585516] Lustre: server umount lustre-OST0000 complete [ 1433.066265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1433.075262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.096759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.103400] Lustre: Skipped 1 previous similar message [ 1436.128756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.034845] Lustre: server umount lustre-MDT0000 complete [ 1441.986870] LustreError: 45997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971396 with bad export cookie 17782771600071644956 [ 1441.988050] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.004822] LustreError: 45997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1442.652683] Lustre: server umount lustre-MDT0001 complete [ 1456.205848] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:30:08 (1788971408) [ 1465.559174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.134365] LustreError: 50445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.152451] LustreError: 50445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1466.206170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.459820] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.344258] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1476.584846] LustreError: 50445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.736665] LustreError: 50445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.776375] LustreError: 50445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1483.533727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.866019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1487.937948] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1494.601601] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1500.254828] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.013978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.357500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.092535] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1521.368947] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.860437] LustreError: 52972:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1526.827529] Lustre: Modifying parameter general.timeout=1 in log params [ 1528.528172] Lustre: Modifying parameter general.timeout=3 in log params [ 1528.538023] Lustre: Skipped 1 previous similar message [ 1531.107918] Lustre: Modifying parameter general.timeout=6 in log params [ 1531.113561] Lustre: Skipped 2 previous similar messages [ 1535.765734] Lustre: Modifying parameter general.timeout=11 in log params [ 1535.769362] Lustre: Skipped 4 previous similar messages [ 1544.186681] Lustre: Modifying parameter general.timeout=20 in log params [ 1544.198249] Lustre: Skipped 8 previous similar messages [ 1561.227631] Lustre: Modifying parameter general.timeout=37 in log params [ 1561.236726] Lustre: Skipped 16 previous similar messages [ 1593.661537] Lustre: Modifying parameter general.timeout=69 in log params [ 1593.663936] Lustre: Skipped 31 previous similar messages [ 1657.674831] Lustre: Disabling parameter general.timeout= in log params [ 1657.683928] Lustre: Skipped 60 previous similar messages [ 1661.300318] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:33:34 (1788971614) [ 1664.355620] Lustre: 59920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1666.337564] Lustre: 60019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1670.046783] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1670.064110] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1674.204358] Lustre: 60442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1674.218289] Lustre: 60442:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1708.555818] Lustre: 61806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1708.576827] LustreError: 61806:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1712.758342] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:34:25 (1788971665) [ 1728.993025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.998636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1729.016504] Lustre: Skipped 1 previous similar message [ 1729.035787] Lustre: Skipped 4 previous similar messages [ 1731.805863] Lustre: server umount lustre-MDT0000 complete [ 1734.113403] LustreError: 55071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.135434] LustreError: 55071:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1735.195294] LustreError: 53566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971689 with bad export cookie 17782771600071645838 [ 1735.200974] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1735.204127] LustreError: 53566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1735.525642] Lustre: server umount lustre-MDT0001 complete [ 1748.076933] Lustre: server umount lustre-OST0000 complete [ 1771.634308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.047363] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1772.393154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.692544] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1788.763084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.927662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1789.000144] Lustre: 64339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.084934] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1801.639078] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1807.937821] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1815.582389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.758645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1816.113160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.121061] Lustre: Skipped 1 previous similar message [ 1822.018495] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1830.532441] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1832.456905] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1832.471351] Lustre: Skipped 23 previous similar messages [ 1832.542565] Lustre: 66905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1843.683679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.685207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1843.706771] Lustre: Skipped 1 previous similar message [ 1843.723924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.750670] Lustre: Skipped 1 previous similar message [ 1845.882945] Lustre: server umount lustre-OST0000 complete [ 1850.848585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.927814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.934104] Lustre: Skipped 2 previous similar messages [ 1856.616454] Lustre: server umount lustre-MDT0000 complete [ 1859.040398] LustreError: 65129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.055392] LustreError: 65129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1860.784967] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971815 with bad export cookie 17782771600071648981 [ 1860.799984] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.802703] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1861.246433] Lustre: server umount lustre-MDT0001 complete [ 1873.093826] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1877.121808] Key type lgssc unregistered [ 1877.389607] LNet: 68112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.396915] LNetError: 68112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.407841] LNet: Removed LNI 192.168.202.135@tcp [ 1878.275208] Key type .llcrypt unregistered [ 1878.277326] Key type ._llcrypt unregistered [ 1906.771790] Key type ._llcrypt registered [ 1906.773860] Key type .llcrypt registered [ 1906.861324] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1907.773988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.786244] alg: No test for adler32 (adler32-zlib) [ 1908.820305] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1909.045624] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1910.751281] Key type lgssc registered [ 1911.588949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.426537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1918.467323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.044890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.742256] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1930.214644] LustreError: 69522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.245165] LustreError: 69522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1933.158579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.728280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1937.930099] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1945.876228] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.041135] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1959.363775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.003918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.044484] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1973.707325] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1986.292790] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1998.827758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.838637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.845588] Lustre: Skipped 2 previous similar messages [ 1998.868756] Lustre: Skipped 2 previous similar messages [ 2002.303741] Lustre: server umount lustre-MDT0000 complete [ 2003.945885] LustreError: 69523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.968916] LustreError: 69523:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2005.500852] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971959 with bad export cookie 9498513259912109521 [ 2005.506029] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.506032] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2005.794892] Lustre: server umount lustre-MDT0001 complete [ 2019.127374] Lustre: server umount lustre-OST0000 complete [ 2046.756724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.110886] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2047.158595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2047.466256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.825897] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2062.999198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.137773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2063.199513] Lustre: 74397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2063.854105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.388735] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2078.272485] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2084.678874] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2091.499280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.706695] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2092.000598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.842807] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2106.458965] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2108.472461] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2110.369308] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2119.139255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.146798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.169280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.653609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2121.316948] Lustre: server umount lustre-OST0000 complete [ 2129.890994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.893584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.917531] Lustre: Skipped 1 previous similar message [ 2129.927094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.933816] Lustre: Skipped 2 previous similar messages [ 2135.008667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.014269] Lustre: Skipped 1 previous similar message [ 2139.105386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.108372] Lustre: Skipped 1 previous similar message [ 2140.127374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2140.694481] Lustre: server umount lustre-MDT0000 complete [ 2144.227779] LustreError: 74409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.244878] LustreError: 74409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2144.697217] LustreError: 74388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972098 with bad export cookie 9498513259912110368 [ 2144.706708] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.712859] LustreError: 74388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2145.170223] Lustre: server umount lustre-MDT0001 complete [ 2158.105996] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2160.397170] Key type lgssc unregistered [ 2160.663296] LNet: 78185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.668630] LNetError: 78185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.685849] LNet: Removed LNI 192.168.202.135@tcp [ 2161.386238] Key type .llcrypt unregistered [ 2161.388400] Key type ._llcrypt unregistered [ 2185.845928] Key type ._llcrypt registered [ 2185.849488] Key type .llcrypt registered [ 2185.972507] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2187.489581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.532607] alg: No test for adler32 (adler32-zlib) [ 2188.518578] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2188.737373] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2190.447635] Key type lgssc registered [ 2191.736744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.913683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.941490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.601612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.346598] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2210.794298] LustreError: 79594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.824501] LustreError: 79594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2215.306061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.136206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.685666] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2230.034619] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.715657] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2245.401714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.359417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.901235] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2264.738194] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2272.146973] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2273.175987] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2275.006581] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2275.013262] Lustre: Skipped 1 previous similar message [ 2277.933204] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2277.938168] Lustre: Skipped 2 previous similar messages [ 2282.730529] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2282.734460] Lustre: Skipped 4 previous similar messages [ 2291.226378] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2291.232644] Lustre: Skipped 8 previous similar messages [ 2308.326176] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2308.331964] Lustre: Skipped 17 previous similar messages [ 2340.349536] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2340.355115] Lustre: Skipped 35 previous similar messages [ 2377.434914] Lustre: 87698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2380.605934] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:45:33 (1788972333) [ 2383.310660] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2386.220204] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:45:38 (1788972338) [ 2393.065927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.068359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.082738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.111961] Lustre: Skipped 1 previous similar message [ 2398.188925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.211114] Lustre: Skipped 2 previous similar messages [ 2403.306883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2403.320623] Lustre: Skipped 1 previous similar message [ 2406.368050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2406.547683] Lustre: server umount lustre-OST0000 complete [ 2413.537560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.554596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.590339] Lustre: Skipped 1 previous similar message [ 2413.618404] Lustre: Skipped 1 previous similar message [ 2417.760693] Lustre: server umount lustre-MDT0000 complete [ 2418.657390] LustreError: 80324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.677072] LustreError: 80324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2422.431541] LustreError: 82679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972376 with bad export cookie 10561970149991806703 [ 2422.460351] LustreError: 82679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2422.467519] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.217314] Lustre: server umount lustre-MDT0001 complete [ 2433.732562] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2436.677138] Key type lgssc unregistered [ 2436.910976] LNet: 88947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.922723] LNetError: 88947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.941877] LNet: Removed LNI 192.168.202.135@tcp [ 2437.559277] Key type .llcrypt unregistered [ 2437.561656] Key type ._llcrypt unregistered [ 2442.524756] Key type ._llcrypt registered [ 2442.532383] Key type .llcrypt registered [ 2442.678658] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2449.804379] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules [ 2450.964849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2451.069059] alg: No test for adler32 (adler32-zlib) [ 2452.148672] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2452.334957] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2454.057205] Key type lgssc registered [ 2455.297807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2456.372992] LustreError: 90194:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2474.850645] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2478.897169] LustreError: 90194:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2485.412442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.424216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.920551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.503353] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2497.507611] LustreError: 91219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.538324] LustreError: 91219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2502.365969] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:47:35 (1788972455) [ 2502.642750] LustreError: 91220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.352437] LustreError: 91251:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2507.358992] LustreError: 91251:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2507.369979] LustreError: 91251:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2509.702732] Lustre: server umount lustre-MDT0000 complete [ 2523.736434] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2527.277775] Key type lgssc unregistered [ 2527.648360] LNet: 92745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.655920] LNetError: 92745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2527.691775] LNet: Removed LNI 192.168.202.135@tcp [ 2528.993942] Key type .llcrypt unregistered [ 2528.996738] Key type ._llcrypt unregistered [ 2552.323393] Key type ._llcrypt registered [ 2552.326826] Key type .llcrypt registered [ 2552.435544] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2554.109551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2554.190289] alg: No test for adler32 (adler32-zlib) [ 2555.332888] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2555.670244] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2557.464612] Key type lgssc registered [ 2558.455481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2567.113377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2567.129121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.875126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.242374] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2578.404610] LustreError: 93963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.418764] LustreError: 93963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2581.710151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.361311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2587.579854] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2594.982143] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2601.143979] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2608.563609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.163439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.138801] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2622.731558] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2674.568623] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:50:27 (1788972627) [ 2677.318760] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2679.499905] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:50:32 (1788972632) [ 2688.994755] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.008885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.023521] Lustre: Skipped 1 previous similar message [ 2689.041192] Lustre: Skipped 2 previous similar messages [ 2691.428873] Lustre: server umount lustre-MDT0000 complete [ 2694.112895] LustreError: 93963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.127493] LustreError: 93963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2695.572276] LustreError: 93944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972649 with bad export cookie 15188512456695599098 [ 2695.594203] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.893482] Lustre: server umount lustre-MDT0001 complete [ 2710.496288] Lustre: server umount lustre-OST0000 complete [ 2725.945879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.329223] LustreError: 98237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.504126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.003811] LustreError: 98238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.519895] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2736.104105] LustreError: 98237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.000829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.561909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2747.997204] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2756.469696] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2763.194743] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2777.866628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.126635] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2778.149299] LustreError: 98227:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2778.195164] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2778.206851] LustreError: 100321:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2778.223760] LustreError: 100321:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2778.228663] LustreError: 100321:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2778.346503] Lustre: server umount lustre-OST0000 complete [ 2778.352643] LustreError: 100321:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2788.257037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.490224] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2788.500685] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2788.553396] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2788.565055] LustreError: 100789:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2788.575162] LustreError: 100789:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2788.580989] LustreError: 100789:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2788.669639] Lustre: server umount lustre-OST0000 complete [ 2788.673473] LustreError: 100789:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2799.061446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.310080] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2799.319561] Lustre: Found index 0 for lustre-OST0000, updating log [ 2799.333532] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2799.395635] Lustre: lustre-OST0000: new disk, initializing [ 2799.399335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2799.405060] Lustre: 101322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2799.780331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.578972] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2810.877515] Lustre: Failing over lustre-OST0000 [ 2811.171496] Lustre: server umount lustre-OST0000 complete [ 2815.209726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.219789] Lustre: Skipped 1 previous similar message [ 2815.225566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.576600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.584139] Lustre: Skipped 2 previous similar messages [ 2821.505981] Lustre: server umount lustre-MDT0000 complete [ 2825.699066] LustreError: 98237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.739893] LustreError: 98237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2826.149729] LustreError: 98219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972780 with bad export cookie 15188512456695601842 [ 2826.151738] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.166518] LustreError: 98219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2826.439850] Lustre: server umount lustre-MDT0001 complete [ 2837.352211] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:53:10 (1788972790) [ 2859.532613] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2875.577950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.992504] LustreError: 103690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.112185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.783973] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2889.966576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.506469] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2899.229633] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2907.901521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.676496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.682504] Lustre: Skipped 1 previous similar message [ 2916.809113] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2926.703756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.917651] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2927.101160] Lustre: lustre-OST0001: new disk, initializing [ 2927.110370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2927.130401] Lustre: 106007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2934.626994] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2936.404737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2936.424399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2936.606624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2942.386685] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2950.210227] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2957.283686] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.287912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2957.313122] Lustre: Skipped 3 previous similar messages [ 2957.329379] Lustre: Skipped 2 previous similar messages [ 2961.824296] Lustre: server umount lustre-MDT0001 complete [ 2962.404602] LustreError: 103686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.433366] LustreError: 103686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2973.962814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.041981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2974.047137] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2974.094146] Lustre: 103680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2974.777693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2974.787236] Lustre: Skipped 1 previous similar message [ 2975.306412] LustreError: 107538:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2975.329853] LustreError: 107538:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2975.341309] LustreError: 103682:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2975.504653] LustreError: 107548:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.135@tcp: cfg command failed: rc = -17 [ 2975.520343] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2975.520343] [ 2979.745854] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2979.840503] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2997.086506] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 12:55:49 (1788972949) [ 3020.151817] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3042.554490] Lustre: 109624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3070.241597] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3073.970165] Lustre: 110761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3184.608115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.611176] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.611783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.611787] Lustre: Skipped 1 previous similar message [ 3184.655591] Lustre: Skipped 3 previous similar messages [ 3186.002982] Lustre: server umount lustre-MDT0000 complete [ 3189.730859] LustreError: 103686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.750900] LustreError: 103686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 3190.614636] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973144 with bad export cookie 15188512456695602836 [ 3190.617538] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.635545] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3191.062592] Lustre: server umount lustre-MDT0001 complete [ 3205.942122] Lustre: server umount lustre-OST0000 complete [ 3221.003471] Lustre: server umount lustre-OST0001 complete [ 3229.521218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3240.982781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3250.211346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3261.506903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3281.426381] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3292.573860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.597038] Lustre: lustre-MDT0000: reset Object Index mappings [ 3293.597580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.414135] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3310.639463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.661695] Lustre: lustre-MDT0001: reset Object Index mappings [ 3316.757289] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3320.406469] Lustre: 115888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3332.436893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.769791] LustreError: 116243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.806610] LustreError: 116243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3336.814270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3338.806725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3343.994728] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3357.094220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.844845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3363.858682] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3367.529911] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3380.430844] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3385.326143] Lustre: 117768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3445.588526] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:03:18 (1788973398) [ 3469.797856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3469.809573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.829400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.835523] Lustre: Skipped 3 previous similar messages [ 3476.235272] Lustre: server umount lustre-MDT0000 complete [ 3480.298101] LustreError: 114707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973434 with bad export cookie 15188512456695649701 [ 3480.302269] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.321908] LustreError: 114707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3481.571802] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.595860] Lustre: Skipped 3 previous similar messages [ 3486.882536] Lustre: server umount lustre-MDT0001 complete [ 3491.316067] LustreError: 118528:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x6f:0x0].0x0 (ffff9439fbd7b500) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3497.805667] Lustre: server umount lustre-OST0000 complete [ 3516.383141] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3517.052797] Lustre: server umount lustre-OST0001 complete [ 3525.449829] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 3542.116248] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3593.973384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.340936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3594.395971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3594.494597] Lustre: lustre-MDT0000: new disk, initializing [ 3594.557307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.563399] Lustre: Skipped 3 previous similar messages [ 3594.574160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3600.625617] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3608.544370] Lustre: server umount lustre-MDT0000 complete [ 3619.331265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.843371] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3619.858151] Lustre: 122064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3619.880045] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3625.191210] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3631.616158] Lustre: server umount lustre-MDT0000 complete [ 3646.522047] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:06:38 (1788973598) [ 3649.481133] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3651.875748] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:06:44 (1788973604) [ 3670.439808] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3682.263431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.838453] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3697.242847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.437159] Lustre: 124116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3697.549435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3697.556206] Lustre: Skipped 1 previous similar message [ 3697.673516] Lustre: lustre-MDT0001: new disk, initializing [ 3697.911529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3697.930715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3703.175663] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3707.713895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3716.777482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.310145] Lustre: lustre-OST0000: new disk, initializing [ 3717.312818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3717.353637] Lustre: 125693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3718.811711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3718.828342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3718.959123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3726.005515] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3737.831066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3737.986811] Lustre: lustre-OST0001: new disk, initializing [ 3737.994789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3738.002826] Lustre: 126564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.154411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3740.163119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3740.313847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3745.123140] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3757.003716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3761.922812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4178.253561] Lustre: 132982:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4178.266838] Lustre: 132982:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4178.273511] Lustre: 132982:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4178.283396] Lustre: 132982:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4178.289801] Lustre: 132982:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4178.295021] Lustre: 132982:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4491.553631] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:20:43 (1788974443) [ 4508.131142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.137576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4508.141350] Lustre: Skipped 2 previous similar messages [ 4508.145473] Lustre: Skipped 8 previous similar messages [ 4513.271428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.291603] Lustre: Skipped 6 previous similar messages [ 4514.327141] Lustre: server umount lustre-MDT0000 complete [ 4518.097237] LustreError: 124107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974472 with bad export cookie 15188512456695719134 [ 4518.113253] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.118197] LustreError: 124107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4518.368347] LustreError: 127370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.370216] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.377506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4518.398176] LustreError: 127370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 4518.440249] Lustre: Skipped 3 previous similar messages [ 4524.713372] Lustre: server umount lustre-MDT0001 complete [ 4535.412846] Lustre: server umount lustre-OST0000 complete [ 4546.373587] Lustre: server umount lustre-OST0001 complete [ 4555.455753] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4568.535863] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4610.639681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.875271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4610.906715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4610.969810] Lustre: lustre-MDT0000: new disk, initializing [ 4611.067769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.076428] Lustre: Skipped 5 previous similar messages [ 4611.099188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.587579] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4629.866091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.052707] Lustre: 136046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4630.130835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4630.139123] Lustre: Skipped 1 previous similar message [ 4630.264351] Lustre: lustre-MDT0001: new disk, initializing [ 4630.398219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4630.418792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4635.387756] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4646.192063] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4653.163433] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4660.997780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.402310] Lustre: lustre-OST0000: new disk, initializing [ 4661.410338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4661.417342] Lustre: 138016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4663.170712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4663.192190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4663.348061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4669.801396] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4682.656800] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.635129] Lustre: lustre-MDD0000: changelog on [ 4696.543239] Lustre: lustre-MDD0001: changelog on [ 6716.835397] Lustre: lustre-MDD0001: changelog off [ 6718.839972] Lustre: 136055:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6719.918227] Lustre: lustre-MDD0000: changelog off [ 6726.630284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.634697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.659772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.669686] Lustre: Skipped 4 previous similar messages [ 6731.755772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.758854] Lustre: Skipped 1 previous similar message [ 6732.648227] Lustre: server umount lustre-OST0000 complete [ 6736.864419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.865834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6736.886896] Lustre: Skipped 2 previous similar messages [ 6736.897655] Lustre: Skipped 1 previous similar message [ 6741.986028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6741.991672] Lustre: Skipped 1 previous similar message [ 6742.931929] Lustre: server umount lustre-MDT0000 complete [ 6746.879646] LustreError: 136039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976701 with bad export cookie 15188512456696539793 [ 6746.892089] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6746.912176] LustreError: 136039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6747.748815] Lustre: server umount lustre-MDT0001 complete [ 6760.118840] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 6763.897049] Key type lgssc unregistered [ 6764.398433] LNet: 147175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6764.424627] LNetError: 147175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6764.441288] LNet: Removed LNI 192.168.202.135@tcp [ 6765.648270] Key type .llcrypt unregistered [ 6765.650349] Key type ._llcrypt unregistered [ 6799.139495] Key type ._llcrypt registered [ 6799.142585] Key type .llcrypt registered [ 6799.348726] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 6817.123207] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 6818.158929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6818.230491] alg: No test for adler32 (adler32-zlib) [ 6819.316759] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6819.608294] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 6821.287270] Key type lgssc registered [ 6822.277716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6859.853950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6859.888892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.379931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6861.517031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6861.594426] Lustre: lustre-MDT0000: new disk, initializing [ 6861.660289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.694253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6865.755478] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6878.164279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.362971] Lustre: 150384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.444293] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6878.448845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6878.455643] Lustre: Skipped 1 previous similar message [ 6878.544115] Lustre: lustre-MDT0001: new disk, initializing [ 6878.659294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.723315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6878.738288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6884.515693] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6895.595796] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6902.543388] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6910.454681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.872445] Lustre: lustre-OST0000: new disk, initializing [ 6910.877974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6910.889964] Lustre: 152365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6910.993319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.424656] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 6920.280504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6920.294796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6920.435291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6928.174793] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6934.398761] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_module obdecho/obdecho [ 6936.238666] LustreError: 153367:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6936.246715] LustreError: 153367:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6938.335093] LustreError: 153463:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6955.489452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6955.495691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.511026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.514119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.519327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.530967] Lustre: Skipped 1 previous similar message [ 6958.984806] Lustre: server umount lustre-MDT0000 complete [ 6962.649437] LustreError: 150377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976916 with bad export cookie 11332818007674877395 [ 6962.658536] LustreError: 150377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6962.658752] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6962.977874] Lustre: server umount lustre-MDT0001 complete [ 6977.591452] Lustre: server umount lustre-OST0000 complete [ 7001.562407] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 7016.983789] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7064.530541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.808519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7064.836108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7064.925893] Lustre: lustre-MDT0000: new disk, initializing [ 7065.020608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.035819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7070.082394] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7081.483428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.603406] Lustre: 157203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7081.668536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7081.677192] Lustre: Skipped 1 previous similar message [ 7081.755277] Lustre: lustre-MDT0001: new disk, initializing [ 7081.892586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7081.936215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7081.964016] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7086.735168] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7097.028225] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7102.780268] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7109.654479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7109.933651] Lustre: lustre-OST0000: new disk, initializing [ 7109.938479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7109.946255] Lustre: 159173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7110.040211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.360326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7116.382937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7116.497837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7116.618109] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7127.636809] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7132.930902] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7133.222748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7139.868722] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7140.217357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7147.489076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7147.497678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.503987] LustreError: Skipped 1 previous similar message [ 7147.522381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7147.530769] Lustre: Skipped 3 previous similar messages [ 7152.609920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.615433] Lustre: Skipped 1 previous similar message [ 7157.215308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7157.459267] Lustre: server umount lustre-OST0000 complete [ 7162.849691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.853593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7162.864967] Lustre: Skipped 2 previous similar messages [ 7162.884362] Lustre: Skipped 1 previous similar message [ 7167.970496] LustreError: 157215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.981606] LustreError: 157215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7167.995953] Lustre: server umount lustre-MDT0000 complete [ 7172.411408] LustreError: 157194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977126 with bad export cookie 11332818007674878354 [ 7172.427308] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7172.429253] LustreError: 157194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7172.815160] Lustre: server umount lustre-MDT0001 complete [ 7177.186844] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:05:29 (1788977129) [ 7179.140186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7181.319077] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:05:34 (1788977134) [ 7190.893831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.382597] LustreError: 161587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.517542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.645637] LustreError: 161588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.703441] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7201.767476] LustreError: 161587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.334742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.849494] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7220.742184] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7228.372651] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.667547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.349994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.357405] Lustre: Skipped 1 previous similar message [ 7245.385873] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7255.236730] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.396186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.406651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.417675] Lustre: Skipped 1 previous similar message [ 7271.435701] Lustre: Skipped 2 previous similar messages [ 7273.241634] Lustre: server umount lustre-MDT0000 complete [ 7276.522754] LustreError: 161582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.536706] LustreError: 161582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7278.879648] LustreError: 161569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977233 with bad export cookie 11332818007674879236 [ 7278.901553] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.906235] LustreError: 161569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7281.633891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.651066] Lustre: Skipped 1 previous similar message [ 7285.646976] Lustre: server umount lustre-MDT0001 complete [ 7294.106239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7308.546625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.820806] LustreError: 165456:0:(llog.c:1655:llog_backup()) MGC192.168.202.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7308.827551] Lustre: 165456:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7309.281357] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.309463] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7327.939903] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.955261] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 7359.974890] LustreError: 165463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7360.011281] LustreError: 165463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 7369.696301] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7369.704048] Lustre: 165456:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7410.682978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.805428] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7415.933193] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7423.976383] LustreError: 165490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.993689] LustreError: 165490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 43 previous similar messages [ 7426.243502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.341678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7449.588636] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9d4641ef239c965d [ 7449.606878] Lustre: MGC192.168.202.135@tcp: Connection restored to 0@lo (at 0@lo) [ 7455.225390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7457.170680] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7466.944974] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7468.769101] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7476.434532] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7478.433524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7506.219611] Lustre: server umount lustre-MDT0000 complete [ 7506.401975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7506.418241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.464392] Lustre: Skipped 2 previous similar messages [ 7510.517284] LustreError: 161567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977464 with bad export cookie 11332818007674885725 [ 7510.519417] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7510.524969] LustreError: 161567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7510.872915] Lustre: server umount lustre-MDT0001 complete [ 7525.507064] Lustre: server umount lustre-OST0000 complete [ 7548.247738] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 7559.385891] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7603.602862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.017119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7604.074908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7604.205338] Lustre: lustre-MDT0000: new disk, initializing [ 7604.318210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7604.323213] Lustre: Skipped 1 previous similar message [ 7604.366775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7610.352894] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7623.723810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.943434] Lustre: 170566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7624.281539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7630.357584] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7643.877910] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.708160] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7659.127482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.414811] Lustre: lustre-OST0000: new disk, initializing [ 7659.417273] Lustre: Skipped 1 previous similar message [ 7659.421305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7659.424734] Lustre: Skipped 2 previous similar messages [ 7659.429441] Lustre: 172534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7665.243918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7665.263350] Lustre: Skipped 1 previous similar message [ 7665.277341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7665.512369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7667.029195] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7677.983708] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7684.818546] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7685.157551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7691.463935] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7691.832842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7696.352897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.353402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.358424] LustreError: Skipped 1 previous similar message [ 7696.359028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.359032] Lustre: Skipped 4 previous similar messages [ 7696.418669] Lustre: Skipped 1 previous similar message [ 7700.757393] Lustre: server umount lustre-OST0000 complete [ 7706.080881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7711.868520] Lustre: server umount lustre-MDT0000 complete [ 7715.811429] LustreError: 170576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.839544] LustreError: 170576:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 7716.398763] LustreError: 170557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977670 with bad export cookie 11332818007674902539 [ 7716.412487] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.421945] LustreError: 170557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.878802] Lustre: server umount lustre-MDT0001 complete [ 7726.911643] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:14:39 (1788977679) [ 7733.871459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.984939] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7750.343133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.446585] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7765.720949] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7772.963334] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7782.141737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7789.329938] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7797.419654] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7820.137048] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:16:13 (1788977773) [ 7824.888648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.898735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7824.907267] Lustre: Skipped 2 previous similar messages [ 7824.918548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7824.965875] Lustre: Skipped 6 previous similar messages [ 7830.516674] Lustre: server umount lustre-OST0000 complete [ 7841.383779] Lustre: server umount lustre-MDT0000 complete [ 7846.223433] LustreError: 174931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977800 with bad export cookie 11332818007674903421 [ 7846.262490] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7846.971775] Lustre: server umount lustre-MDT0001 complete [ 7860.455893] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 7863.100367] Key type lgssc unregistered [ 7863.454550] LNet: 178700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7863.482075] LNetError: 178700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7863.497955] LNet: Removed LNI 192.168.202.135@tcp [ 7864.329351] Key type .llcrypt unregistered [ 7864.331203] Key type ._llcrypt unregistered [ 7886.602835] Key type ._llcrypt registered [ 7886.604928] Key type .llcrypt registered [ 7886.697770] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7887.688260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7887.758772] alg: No test for adler32 (adler32-zlib) [ 7888.971875] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7889.165046] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 7890.823273] Key type lgssc registered [ 7892.277589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7900.627966] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7900.654452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.386688] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7921.569542] LustreError: 179940:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 7926.687814] LustreError: 179940:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7926.713417] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7926.716898] LustreError: MGC192.168.202.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7926.731292] LustreError: 179940:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7926.744349] LustreError: 179940:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7926.752536] LustreError: 179940:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7926.832750] Lustre: server umount lustre-OST0000 complete [ 7926.842832] LustreError: 179940:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7937.655601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.220988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.493370] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7948.840078] LustreError: 180438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.883623] LustreError: 180438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7953.889873] LustreError: 180439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.090672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.626148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7961.205562] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7970.100144] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.312406] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7985.931499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.313223] LustreError: 182259:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7986.331645] Lustre: 182259:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7986.642343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.882369] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8005.251944] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8010.219341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.248099] Lustre: Skipped 2 previous similar messages [ 8010.250298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.281754] Lustre: Skipped 2 previous similar messages [ 8014.477646] Lustre: server umount lustre-MDT0000 complete [ 8015.333410] LustreError: 182701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.366584] LustreError: 182701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8018.583359] LustreError: 180421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977972 with bad export cookie 1795586696372788318 [ 8018.588552] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8018.595660] LustreError: 180421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8018.921521] Lustre: server umount lustre-MDT0001 complete [ 8034.122968] Lustre: server umount lustre-OST0000 complete [ 8050.709579] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8067.690841] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8113.082081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.356394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8113.391689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8113.495623] Lustre: lustre-MDT0000: new disk, initializing [ 8113.624281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.653923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8118.583900] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8131.049153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.181792] Lustre: 185807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8131.261961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8131.264413] Lustre: Skipped 1 previous similar message [ 8131.358861] Lustre: lustre-MDT0001: new disk, initializing [ 8131.463405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8131.553401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8131.569884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8137.187096] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8148.279535] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8155.244067] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8165.615183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.064802] Lustre: lustre-OST0000: new disk, initializing [ 8166.074358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8166.079695] Lustre: 187779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8166.224310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.675183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8174.684196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8174.783692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8175.412226] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8188.446633] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.615847] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8194.825761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8201.614944] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8201.840248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8205.799126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8205.800274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.805250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8205.824561] LustreError: Skipped 1 previous similar message [ 8205.885578] Lustre: Skipped 1 previous similar message [ 8210.743277] Lustre: server umount lustre-OST0000 complete [ 8216.046760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.072663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8216.075536] Lustre: Skipped 1 previous similar message [ 8221.158311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.172358] Lustre: Skipped 1 previous similar message [ 8226.274646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8226.293164] Lustre: Skipped 1 previous similar message [ 8229.347103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8230.147283] Lustre: server umount lustre-MDT0000 complete [ 8231.392473] LustreError: 185819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.403717] LustreError: 185819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8235.157591] LustreError: 185799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978189 with bad export cookie 1795586696372789032 [ 8235.179853] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8235.918911] Lustre: server umount lustre-MDT0001 complete [ 8248.250197] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:23:20 (1788978200) [ 8270.287765] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8274.579645] Key type lgssc unregistered [ 8275.193824] LNet: 190729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8275.206146] LNetError: 190729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8276.264181] LNet: Removed LNI 192.168.202.135@tcp [ 8277.346991] Key type .llcrypt unregistered [ 8277.351253] Key type ._llcrypt unregistered [ 8303.843506] Key type ._llcrypt registered [ 8303.848501] Key type .llcrypt registered [ 8303.954905] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8304.824819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8304.859307] alg: No test for adler32 (adler32-zlib) [ 8306.148719] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8306.334154] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8307.967257] Key type lgssc registered [ 8309.157720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8320.389677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8320.429465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.957854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.530951] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8332.257143] LustreError: 191945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.279855] LustreError: 191945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8336.835072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.299592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.132507] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8350.205922] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8356.511784] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8365.175767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8365.349445] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8365.352694] LustreError: 193808:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8365.359294] Lustre: 193808:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8365.455990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8372.298622] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8381.417609] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8387.042526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8387.046505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.062626] Lustre: Skipped 2 previous similar messages [ 8387.079266] Lustre: Skipped 2 previous similar messages [ 8389.879084] Lustre: server umount lustre-MDT0000 complete [ 8392.164838] LustreError: 192948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8392.192641] LustreError: 192948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8394.432563] LustreError: 191927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978348 with bad export cookie 3711162753147718353 [ 8394.473844] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8395.295453] LustreError: 192674:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000045779897 x1875879667986688/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8395.497393] Lustre: server umount lustre-MDT0001 complete [ 8410.455907] Lustre: server umount lustre-OST0000 complete [ 8423.823757] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8435.963897] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8478.164157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8478.373261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8478.396059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8478.457487] Lustre: lustre-MDT0000: new disk, initializing [ 8478.586270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.626124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8484.047377] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8497.398912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.681881] Lustre: 197334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8497.822159] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8497.828034] Lustre: Skipped 1 previous similar message [ 8497.838173] LustreError: 198117:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8497.849361] LustreError: 198117:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8497.866587] Lustre: 198117:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8497.898657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8497.908943] Lustre: Skipped 1 previous similar message [ 8498.155389] Lustre: lustre-MDT0001: new disk, initializing [ 8498.516416] Lustre: 198117:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-client: rc = -28 [ 8498.552531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8498.601429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8498.614978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8505.443653] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8518.455611] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8528.230872] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8539.845832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.368221] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8540.376056] Lustre: Skipped 1 previous similar message [ 8540.382499] LustreError: 199281:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8540.400843] LustreError: 199281:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8540.417873] Lustre: 199281:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8540.485698] Lustre: lustre-OST0000: new disk, initializing [ 8540.497498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8540.518986] Lustre: 199304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8540.914448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.026123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8549.042670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8549.412502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8552.733241] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8565.388715] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8571.031928] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8571.336870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8576.772337] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8577.060960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8580.068110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.088190] Lustre: Skipped 1 previous similar message [ 8580.089243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8580.095508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8580.124614] Lustre: Skipped 1 previous similar message [ 8585.202645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.210808] Lustre: Skipped 1 previous similar message [ 8585.529677] Lustre: server umount lustre-OST0000 complete [ 8590.305453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.320444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8590.334674] Lustre: Skipped 1 previous similar message [ 8595.424807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.434803] Lustre: Skipped 1 previous similar message [ 8595.973283] Lustre: server umount lustre-MDT0000 complete [ 8600.418937] LustreError: 197329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978554 with bad export cookie 3711162753147719067 [ 8600.432864] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.443644] LustreError: 197329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8600.995901] Lustre: server umount lustre-MDT0001 complete [ 8612.018496] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:29:24 (1788978564) [ 8634.752747] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8646.354755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.875423] LustreError: 202439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.940821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.262823] LustreError: 202440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.744094] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8657.378977] LustreError: 202439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.274211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.807088] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8674.006307] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8682.978680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8683.637923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.648735] Lustre: Skipped 1 previous similar message [ 8691.756530] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8703.069666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.222691] Lustre: lustre-OST0001: new disk, initializing [ 8703.225988] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8703.231349] Lustre: 204748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8710.854749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8710.878433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8711.166696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8711.206437] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8724.914820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8735.237499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8738.357847] Lustre: DEBUG MARKER: ADD OST3 [ 8746.076934] loop: module loaded [ 8746.182711] loop0: detected capacity change from 0 to 204800000 [ 8746.465661] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8746.486303] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8746.522057] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8754.072652] Lustre: DEBUG MARKER: STOP OST3 [ 8754.942869] loop0: detected capacity change from 0 to 204800000 [ 8762.912700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8763.160257] Lustre: lustre-OST0003: new disk, initializing [ 8763.177194] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8763.212216] Lustre: 206833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8763.530520] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8763.539870] Lustre: Skipped 1 previous similar message [ 8765.044703] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8765.255162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8765.278075] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8765.341491] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8766.598028] Lustre: Failing over lustre-OST0003 [ 8767.123193] LustreError: 206923:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8767.210862] Lustre: server umount lustre-OST0003 complete [ 8769.823037] LustreError: 203927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.855691] LustreError: 203927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8770.560336] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8770.569186] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.612576] Lustre: Skipped 2 previous similar messages [ 8775.105561] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8785.936671] LustreError: 207545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.991876] LustreError: 207545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 8791.211670] Lustre: DEBUG MARKER: START OST3 again [ 8799.749862] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.152196] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8800.461270] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8801.820045] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8801.825196] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8808.026061] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8834.379118] Lustre: Failing over lustre-OST0003 [ 8834.591329] Lustre: server umount lustre-OST0003 complete [ 8836.082572] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.106053] LustreError: 203926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.133213] LustreError: 203926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 8853.477327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8853.492125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.506937] Lustre: Skipped 1 previous similar message [ 8853.515610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.047908] Lustre: server umount lustre-MDT0000 complete [ 8863.912542] LustreError: 205574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978818 with bad export cookie 3711162753147719956 [ 8863.926479] LustreError: 205574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8863.928347] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8870.569765] Lustre: server umount lustre-MDT0001 complete [ 8875.280943] Lustre: server umount lustre-OST0000 complete [ 8879.642865] Lustre: server umount lustre-OST0001 complete [ 8887.358651] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8897.563599] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8943.578754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.895737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8943.922253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8944.052412] Lustre: lustre-MDT0000: new disk, initializing [ 8944.156088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.160817] Lustre: Skipped 1 previous similar message [ 8944.170262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8949.836368] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8966.096524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.284967] Lustre: 212370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8966.370173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8966.373310] Lustre: Skipped 1 previous similar message [ 8966.478029] Lustre: lustre-MDT0001: new disk, initializing [ 8966.632758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8966.655737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8972.605154] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8985.466722] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8992.697439] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9000.678236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.953414] Lustre: lustre-OST0000: new disk, initializing [ 9000.957235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9000.963607] Lustre: 214340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9002.214888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9002.233655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9002.374421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9008.879914] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9019.905637] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9025.366089] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9025.563226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9030.353638] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9030.605402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9033.203550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9033.213390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.225974] LustreError: Skipped 1 previous similar message [ 9033.239066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9033.242283] Lustre: Skipped 6 previous similar messages [ 9033.247592] Lustre: Skipped 8 previous similar messages [ 9039.415171] Lustre: server umount lustre-OST0000 complete [ 9048.552488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9058.271452] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9058.489972] Lustre: server umount lustre-MDT0000 complete [ 9058.794909] LustreError: 212380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.829643] LustreError: 212380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 9062.727684] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979016 with bad export cookie 3711162753147722252 [ 9062.735086] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.746109] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9063.141188] Lustre: server umount lustre-MDT0001 complete [ 9067.190292] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:36:59 (1788979019) [ 9091.375901] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9108.207727] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9163.501523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.831110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9163.892110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9163.963728] Lustre: lustre-MDT0000: new disk, initializing [ 9164.086591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9168.930885] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9180.802624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.968897] Lustre: 218983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9181.211095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9187.765632] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9203.538466] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.370585] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9226.370785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9226.661735] Lustre: 220952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9226.776646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.783563] Lustre: Skipped 4 previous similar messages [ 9228.557400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9228.791839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9234.720750] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9248.518912] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9254.161953] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9254.397419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.875509] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9260.260949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9264.616578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9264.624735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9264.638108] Lustre: Skipped 3 previous similar messages [ 9264.640290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9264.670698] Lustre: Skipped 8 previous similar messages [ 9270.232219] Lustre: server umount lustre-OST0000 complete [ 9281.733251] Lustre: server umount lustre-MDT0000 complete [ 9285.089690] LustreError: 218995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.120525] LustreError: 218995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9286.357884] LustreError: 218974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979240 with bad export cookie 3711162753147723134 [ 9286.385172] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9286.747458] Lustre: server umount lustre-MDT0001 complete [ 9295.867524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.318257] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9311.835870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.100221] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9328.205948] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9335.634789] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9346.435762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9356.333880] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9369.508371] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9381.856288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.858068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.922473] Lustre: Skipped 2 previous similar messages [ 9381.966658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9381.981976] Lustre: Skipped 7 previous similar messages [ 9387.506547] Lustre: server umount lustre-OST0000 complete [ 9398.650290] Lustre: server umount lustre-MDT0000 complete [ 9403.533589] LustreError: 223152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979357 with bad export cookie 3711162753147724016 [ 9403.534732] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.552961] LustreError: 223152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9404.193718] Lustre: server umount lustre-MDT0001 complete [ 9512.124880] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 9515.525588] Key type lgssc unregistered [ 9515.803929] LNet: 226836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9515.808963] LNetError: 226836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9515.825065] LNet: Removed LNI 192.168.202.135@tcp [ 9516.634314] Key type .llcrypt unregistered [ 9516.636113] Key type ._llcrypt unregistered [ 9544.596760] Key type ._llcrypt registered [ 9544.601300] Key type .llcrypt registered [ 9544.705272] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9565.363951] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9566.804063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9566.831973] alg: No test for adler32 (adler32-zlib) [ 9567.967189] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 9568.226502] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 9569.919254] Key type lgssc registered [ 9571.366516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9614.488397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9614.533983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.901865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9615.932473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9616.039318] Lustre: lustre-MDT0000: new disk, initializing [ 9616.151206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.187670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9621.598644] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9633.645920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.813884] Lustre: 230191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9633.870634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9633.874430] Lustre: Skipped 1 previous similar message [ 9633.975483] Lustre: lustre-MDT0001: new disk, initializing [ 9634.125750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9634.205510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9634.233821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9639.570337] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9652.386830] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9660.636344] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9669.190498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.480470] Lustre: lustre-OST0000: new disk, initializing [ 9669.492425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9669.505859] Lustre: 232159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9669.618210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9676.538404] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9679.438488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9679.456066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9679.583783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9688.719499] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9695.416263] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9695.869844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9702.727929] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9703.043268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9710.560916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.585879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.607869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9710.617096] Lustre: Skipped 1 previous similar message [ 9711.858537] Lustre: server umount lustre-OST0000 complete [ 9716.192841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9716.205392] LustreError: Skipped 1 previous similar message [ 9716.207207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.227079] Lustre: Skipped 1 previous similar message [ 9716.233242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.805541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.808646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.853081] Lustre: Skipped 1 previous similar message [ 9722.660821] Lustre: server umount lustre-MDT0000 complete [ 9728.378037] LustreError: 230183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979682 with bad export cookie 6534401634356386793 [ 9728.394824] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.230123] Lustre: server umount lustre-MDT0001 complete [ 9762.146394] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9776.317116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.792685] LustreError: 235119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.827095] LustreError: 235119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9776.867456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9782.242952] LustreError: 235120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.330136] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9787.375971] LustreError: 235119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.481517] LustreError: 235120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.301444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9795.684290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9801.213340] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9805.593600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9814.646127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9815.098356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.830568] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9832.447954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9832.663521] Lustre: lustre-OST0001: new disk, initializing [ 9832.668647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9832.674849] Lustre: 237430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9832.762165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.876459] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9842.381746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9842.393419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9842.630863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9855.824950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9865.776129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9889.983784] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:50:41 (1788979841) [ 9892.833660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9892.840308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.876157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9898.986570] LustreError: 236867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.004116] LustreError: 236867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9899.070896] Lustre: server umount lustre-MDT0000 complete [ 9903.692862] LustreError: 237429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979857 with bad export cookie 6534401634356387675 [ 9903.705317] LustreError: 237429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9903.716112] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9904.133718] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.148505] Lustre: Skipped 3 previous similar messages [ 9904.331255] Lustre: server umount lustre-MDT0001 complete [ 9911.466392] Lustre: server umount lustre-OST0000 complete [ 9917.065918] Lustre: server umount lustre-OST0001 complete [ 9925.783482] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9941.343996] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9987.400807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.686615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9987.713238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9987.770630] Lustre: lustre-MDT0000: new disk, initializing [ 9987.828924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9987.851937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9993.881527] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10009.023507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.229762] Lustre: 241420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10009.312112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10009.320742] Lustre: Skipped 1 previous similar message [10009.466630] Lustre: lustre-MDT0001: new disk, initializing [10009.685187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10009.694629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10016.203907] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10028.520763] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10036.931337] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10047.288146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10047.776526] Lustre: lustre-OST0000: new disk, initializing [10047.779211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10047.792282] Lustre: 243394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10049.696644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10049.722930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10049.872109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10056.591569] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10069.700335] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10076.316603] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10076.601764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10082.436682] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10082.735537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10085.866605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10085.872579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10085.881473] Lustre: Skipped 2 previous similar messages [10085.884760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10085.888431] Lustre: Skipped 5 previous similar messages [10096.121800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.130452] Lustre: Skipped 3 previous similar messages [10099.167162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10099.568250] Lustre: server umount lustre-OST0000 complete [10106.337576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.580213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10116.585136] Lustre: Skipped 5 previous similar messages [10118.111930] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10118.510904] Lustre: server umount lustre-MDT0000 complete [10121.698950] LustreError: 241433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.743151] LustreError: 241433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10122.943172] LustreError: 241413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980077 with bad export cookie 6534401634356389810 [10122.964483] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10122.965614] LustreError: 241413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10123.256663] Lustre: server umount lustre-MDT0001 complete [10132.781838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.353454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10133.365556] Lustre: Skipped 2 previous similar messages [10138.608541] LustreError: 245615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10138.636783] LustreError: 245615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10138.988701] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10149.066386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10154.433987] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10164.635516] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10171.786903] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10180.954384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10189.581817] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10199.553748] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10347.490128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10347.511077] LustreError: Skipped 1 previous similar message [10347.522220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10347.558139] Lustre: Skipped 1 previous similar message [10347.567737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10361.311520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10361.645475] Lustre: server umount lustre-OST0000 complete [10380.257320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10380.660077] Lustre: server umount lustre-MDT0000 complete [10382.313681] LustreError: 245610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.331826] LustreError: 245610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10384.768161] LustreError: 245597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980338 with bad export cookie 6534401634356390692 [10384.775102] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10384.786547] LustreError: 245597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10385.182892] Lustre: server umount lustre-MDT0001 complete [10467.063873] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [10470.277389] Key type lgssc unregistered [10470.556494] LNet: 249222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10470.561932] LNetError: 249222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10471.590619] LNet: Removed LNI 192.168.202.135@tcp [10472.630172] Key type .llcrypt unregistered [10472.633060] Key type ._llcrypt unregistered [10502.266415] Key type ._llcrypt registered [10502.269964] Key type .llcrypt registered [10502.391113] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [10521.334100] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10522.678288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10522.973195] alg: No test for adler32 (adler32-zlib) [10524.082356] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10524.364056] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [10526.031941] Key type lgssc registered [10527.841478] Lustre: Echo OBD driver; http://www.lustre.org/ [10587.972889] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10602.594489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10602.636552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.989959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10604.045144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10604.154811] Lustre: lustre-MDT0000: new disk, initializing [10604.350532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10604.379455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10610.128241] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10622.911249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.062984] Lustre: 253324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10623.124107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10623.128520] Lustre: Skipped 1 previous similar message [10623.259275] Lustre: lustre-MDT0001: new disk, initializing [10623.368131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10623.411156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10623.433130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10628.546328] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10634.728521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10642.719834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.303954] Lustre: lustre-OST0000: new disk, initializing [10643.311672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10643.329546] Lustre: 254958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10643.542928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10652.838789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10652.856566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10653.093689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10653.553961] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10668.732622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10669.105555] Lustre: lustre-OST0001: new disk, initializing [10669.116874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10669.128875] Lustre: 255831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10669.322463] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10677.444524] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10677.966213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10677.976520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10678.143210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10691.985909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10700.847240] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10715.626705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10715.636704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10715.656151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.202774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.207044] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.216221] Lustre: Skipped 2 previous similar messages [10719.252711] Lustre: Skipped 2 previous similar messages [10720.831502] Lustre: server umount lustre-MDT0000 complete [10726.347683] LustreError: 254119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980680 with bad export cookie 10310065539976755949 [10726.357319] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10726.364559] LustreError: 254119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10727.011491] Lustre: server umount lustre-MDT0001 complete [10742.958306] Lustre: server umount lustre-OST0000 complete [10745.635061] Lustre: 250809:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788980683/real 1788980683] req@ffff9438c3d74b40 x1875881993785984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788980699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10745.662907] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.848189] Lustre: 250806:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788980685/real 1788980685] req@ffff9438c3d761c0 x1875881993786240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788980701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10747.656981] Lustre: server umount lustre-OST0001 complete [10766.829388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10779.053601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10788.659042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10799.230669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10823.381837] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10835.946644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.980695] Lustre: lustre-MDT0000: reset Object Index mappings [10836.530896] LustreError: 260843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.584955] LustreError: 260843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10836.676388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10841.587996] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10842.093334] LustreError: 260844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.181384] LustreError: 260843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.298961] LustreError: 260844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.722454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.732414] Lustre: lustre-MDT0001: reset Object Index mappings [10852.162162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10856.527240] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10859.866385] Lustre: 261985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10867.181965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.315948] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10877.805085] LustreError: 262340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.879495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10883.621679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.963953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10883.970726] Lustre: Skipped 1 previous similar message [10889.252601] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10890.502849] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10899.285911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10903.254818] Lustre: 263855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10909.668502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.674399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10909.694354] Lustre: Skipped 3 previous similar messages [10909.708456] Lustre: Skipped 4 previous similar messages [10914.809834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10914.821728] Lustre: Skipped 3 previous similar messages [10915.663086] Lustre: server umount lustre-MDT0000 complete [10919.908659] LustreError: 260843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.946466] LustreError: 260843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10920.065344] LustreError: 260823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980874 with bad export cookie 10310065539976762739 [10920.074214] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.084742] LustreError: 260823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10920.486951] Lustre: server umount lustre-MDT0001 complete [10936.585763] Lustre: server umount lustre-OST0000 complete [10950.066262] Lustre: server umount lustre-OST0001 complete [10969.974602] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:08:42 (1788980922) [10987.865883] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [11000.012878] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11044.096797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.441439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11044.487449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11044.719294] Lustre: lustre-MDT0000: new disk, initializing [11044.865745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11044.913582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11050.963977] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11064.320173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.569693] Lustre: 268112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11064.642937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11064.646492] Lustre: Skipped 1 previous similar message [11064.781842] Lustre: lustre-MDT0001: new disk, initializing [11064.984806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11065.018972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11070.691197] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11081.467627] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11089.302483] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11097.441192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.841950] Lustre: lustre-OST0000: new disk, initializing [11097.861088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11097.884724] Lustre: 270080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11098.939430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11098.945514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11099.097055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11105.240261] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11115.431894] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.472412] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11121.703408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11127.910700] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11128.342880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11134.955344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11134.958163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.974521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11135.008727] Lustre: Skipped 1 previous similar message [11145.186793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.188099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [11145.189761] Lustre: Skipped 4 previous similar messages [11145.462346] Lustre: server umount lustre-OST0000 complete [11150.306503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.483388] Lustre: server umount lustre-MDT0000 complete [11160.550042] LustreError: 270132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.566814] LustreError: 270132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11160.885951] LustreError: 268106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981115 with bad export cookie 10310065539976764832 [11160.907720] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11161.497530] Lustre: server umount lustre-MDT0001 complete [11184.256631] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11195.641444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.177584] LustreError: 273019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.226526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11196.230985] Lustre: Skipped 2 previous similar messages [11201.775975] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11211.135396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.273608] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11219.300583] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11226.856146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.527580] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11245.126810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.408666] Lustre: lustre-OST0001: new disk, initializing [11245.415232] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11245.421953] Lustre: 275328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11246.856765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11246.869048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11247.104990] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11253.550391] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11263.423941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11267.251191] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11272.945625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11272.958584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11283.206160] LustreError: 250809:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9438c3d74b40 x1875881993943424/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [11283.424612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11283.434919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11283.444988] Lustre: Skipped 1 previous similar message [11283.456575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11283.468873] Lustre: Skipped 4 previous similar messages [11289.662666] Lustre: server umount lustre-MDT0000 complete [11293.156878] LustreError: 276788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.179429] LustreError: 276788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11293.279331] LustreError: 272999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981247 with bad export cookie 10310065539976765714 [11293.283179] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.291055] LustreError: 272999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11293.538332] Lustre: server umount lustre-MDT0001 complete [11307.564430] Lustre: server umount lustre-OST0000 complete [11322.857483] Lustre: server umount lustre-OST0001 complete [11346.374534] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11358.279958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.725766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11358.733870] Lustre: Skipped 3 previous similar messages [11363.313942] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11374.095813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.848811] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11384.884192] Lustre: 279833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11396.116620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11403.733177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11404.064942] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11414.049410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11419.675196] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11421.596735] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11430.219424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11434.280630] Lustre: 281708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11440.454348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11440.501710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11455.886923] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:16:48 (1788981408) [11475.952055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.958352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.976302] Lustre: Skipped 6 previous similar messages [11475.997855] Lustre: Skipped 6 previous similar messages [11479.177319] Lustre: server umount lustre-MDT0000 complete [11481.071438] LustreError: 278691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.094533] LustreError: 278691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [11483.925518] LustreError: 278671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981438 with bad export cookie 10310065539976772140 [11483.929923] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.933653] LustreError: 278671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11490.364563] Lustre: server umount lustre-MDT0001 complete [11501.327260] Lustre: server umount lustre-OST0000 complete [11505.571798] Lustre: server umount lustre-OST0001 complete [11512.617028] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [11524.363308] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11581.711813] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11595.555812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.940173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11596.002845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11596.123961] Lustre: lustre-MDT0000: new disk, initializing [11596.336159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11601.814589] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11614.898599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11615.067040] Lustre: 286247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11615.322502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11615.335849] Lustre: Skipped 4 previous similar messages [11615.368113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11620.257337] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11634.355614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.649613] Lustre: 287880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11636.930597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11643.040795] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11655.084326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.482477] Lustre: 288753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11657.383982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11657.389709] Lustre: Skipped 1 previous similar message [11657.568686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11662.565755] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11674.051957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11678.702609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11678.707232] Lustre: Skipped 1 previous similar message [11700.893955] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:20:52 (1788981652) [11703.790930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11703.809794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.817407] Lustre: Skipped 4 previous similar messages [11703.837087] Lustre: Skipped 8 previous similar messages [11704.289183] Lustre: server umount lustre-MDT0000 complete [11708.398965] LustreError: 286239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981662 with bad export cookie 10310065539976780050 [11708.400662] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.420049] LustreError: 286239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11709.179512] Lustre: server umount lustre-MDT0001 complete [11714.279552] Lustre: server umount lustre-OST0000 complete [11718.927935] Lustre: server umount lustre-OST0001 complete [11729.201948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.139708] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11740.142712] LustreError: 290998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.172902] LustreError: 290998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11745.998388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.768148] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11761.191403] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11768.716026] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11776.908781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.672747] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11794.037931] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11820.076105] loop1: detected capacity change from 0 to 204800000 [11820.366374] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11820.387627] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11820.417765] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11826.426909] loop1: detected capacity change from 0 to 204800000 [11832.982749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11833.105940] LustreError: 290987:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29054: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11833.120846] LustreError: 290987:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29054 [11833.128457] LustreError: 290987:0:(mgs_handler.c:517:mgs_target_reg()) fs29054-MDT0001: Error checking target index: rc = -13 [11833.154322] LustreError: fs29054-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11833.162663] LustreError: 294719:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11833.178366] LustreError: 294719:0:(tgt_mount.c:2128:server_put_super()) no obd fs29054-MDT0001 [11833.196886] LustreError: 294719:0:(tgt_mount.c:116:server_deregister_mount()) fs29054-MDT0001 not registered [11833.251156] Lustre: server umount fs29054-MDT0001 complete [11833.255849] LustreError: 294719:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11844.670516] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.912426] Lustre: Setting parameter fs29054-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29054-MDT0001 [11845.005597] Lustre: srv-fs29054-MDT0001: No data found on store. Initialize space. [11845.013916] Lustre: Skipped 4 previous similar messages [11845.171682] Lustre: fs29054-MDT0001: new disk, initializing [11845.174273] Lustre: Skipped 3 previous similar messages [11845.342525] Lustre: srv-fs29054-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11845.355606] Lustre: 295185:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29054-MDT0001-mdtlov: can't create new llog: rc = -115 [11845.381735] LustreError: 295185:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11851.547622] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11860.220749] Lustre: server umount fs29054-MDT0001 complete [11871.713979] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11875.415882] loop1: detected capacity change from 0 to 204800000 [11876.860000] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11883.463232] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.191607] Lustre: 296757:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29054-MDT0001-mdtlov: can't create new llog: rc = -115 [11884.208760] LustreError: 296757:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11890.693776] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11896.241481] Lustre: server umount fs29054-MDT0001 complete [11910.113300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11910.124467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11910.131032] Lustre: Skipped 3 previous similar messages [11910.136075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11915.288802] Lustre: server umount lustre-MDT0000 complete [11925.958815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.077084] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11931.653636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11932.119808] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11940.611697] loop1: detected capacity change from 0 to 204800000 [11947.284349] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.098290] Lustre: 299207:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29054-MDT0001-mdtlov: can't create new llog: rc = -115 [11948.101901] LustreError: 299207:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29054-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11953.950203] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11959.233221] Lustre: server umount fs29054-MDT0001 complete [11986.403561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11990.507524] Lustre: server umount lustre-OST0000 complete [12003.432490] Lustre: server umount lustre-MDT0000 complete [12008.239162] LustreError: 299699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981962 with bad export cookie 10310065539976782731 [12008.241917] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.254578] LustreError: 299699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12008.944572] Lustre: server umount lustre-MDT0001 complete [12025.131162] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12029.698754] Key type lgssc unregistered [12030.284721] LNet: 301553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12030.294894] LNetError: 301553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12030.350614] LNet: Removed LNI 192.168.202.135@tcp [12032.079774] Key type .llcrypt unregistered [12032.081071] Key type ._llcrypt unregistered [12058.944440] Key type ._llcrypt registered [12058.949103] Key type .llcrypt registered [12059.108743] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12060.735537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12060.796699] alg: No test for adler32 (adler32-zlib) [12062.134295] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12062.638518] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12064.471327] Key type lgssc registered [12066.327507] Lustre: Echo OBD driver; http://www.lustre.org/ [12076.085353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12076.114421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.789802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12083.184333] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12087.289216] LustreError: 302827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.324175] LustreError: 302827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12092.389451] LustreError: 302826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.738897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.468685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12099.367502] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12108.160451] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12115.226197] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12123.927475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12124.461701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12131.585683] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12138.805422] LustreError: 304659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.434409] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12144.108213] LustreError: 305281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.318060] LustreError: 304659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.348026] LustreError: 304659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12162.535618] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.568619] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12169.973653] loop1: detected capacity change from 0 to 204800000 [12170.165494] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12170.177506] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12170.202175] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12177.611817] loop1: detected capacity change from 0 to 204800000 [12178.653155] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12178.699564] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [12184.142658] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12184.299549] Lustre: fs19560-OST0000: new disk, initializing [12184.302918] Lustre: srv-fs19560-OST0000: No data found on store. Initialize space. [12184.308718] Lustre: 306585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12184.388926] Lustre: fs19560-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12191.557432] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12201.214849] Lustre: server umount fs19560-OST0000 complete [12213.749480] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.797729] LustreError: 304658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [12218.899357] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12220.452201] loop1: detected capacity change from 0 to 204800000 [12223.987386] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12228.949466] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12229.168470] Lustre: fs19560-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.567774] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12250.475536] loop2: detected capacity change from 0 to 204800000 [12250.731411] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12250.747751] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12250.777334] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12258.327910] loop2: detected capacity change from 0 to 204800000 [12266.325105] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12266.468884] LustreError: fs19560-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12266.479559] LustreError: fs19560-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12266.493030] LustreError: 309631:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12266.504126] LustreError: 309631:0:(tgt_mount.c:2128:server_put_super()) no obd fs19560-OST0001 [12266.515084] LustreError: 309631:0:(tgt_mount.c:116:server_deregister_mount()) fs19560-OST0001 not registered [12266.545926] Lustre: server umount fs19560-OST0001 complete [12266.549280] LustreError: 309631:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12279.302211] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12279.549983] Lustre: fs19560-OST0001: new disk, initializing [12279.555926] Lustre: srv-fs19560-OST0001: No data found on store. Initialize space. [12279.569875] Lustre: 310182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12279.679653] Lustre: fs19560-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12280.303620] LustreError: 304657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12280.330169] LustreError: 304657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [12286.912328] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12296.745143] Lustre: server umount fs19560-OST0000 complete [12308.277586] Lustre: server umount fs19560-OST0001 complete [12324.136030] loop1: detected capacity change from 0 to 204800000 [12330.813412] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12331.218563] Lustre: fs19560-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12338.870555] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12347.859316] loop2: detected capacity change from 0 to 204800000 [12355.314129] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.147539] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12372.467505] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12376.800505] Lustre: server umount fs19560-OST0000 complete [12377.611634] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12389.259438] Lustre: server umount fs19560-OST0001 complete [12408.309734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.326818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12408.328000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12408.330075] Lustre: Skipped 1 previous similar message [12413.410081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12413.424284] LustreError: 305281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.426298] Lustre: Skipped 1 previous similar message [12413.473388] LustreError: 305281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 76 previous similar messages [12413.901247] Lustre: server umount lustre-OST0000 complete [12418.528692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.545958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12418.553371] Lustre: Skipped 2 previous similar messages [12423.648475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12423.653316] Lustre: Skipped 1 previous similar message [12423.953680] Lustre: server umount lustre-MDT0000 complete [12429.078535] LustreError: 302806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982383 with bad export cookie 10840470079032584794 [12429.079867] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12429.097915] LustreError: 302806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12429.672636] Lustre: server umount lustre-MDT0001 complete [12442.913570] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12445.800473] Key type lgssc unregistered [12446.243425] LNet: 316107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12446.261233] LNetError: 316107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12446.314262] LNet: Removed LNI 192.168.202.135@tcp [12447.440214] Key type .llcrypt unregistered [12447.443474] Key type ._llcrypt unregistered [12483.661893] Key type ._llcrypt registered [12483.666121] Key type .llcrypt registered [12483.831126] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [12503.291439] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12504.307752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12504.330317] alg: No test for adler32 (adler32-zlib) [12505.712553] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12506.007844] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12507.703414] Key type lgssc registered [12509.063595] Lustre: Echo OBD driver; http://www.lustre.org/ [12548.808352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12548.845211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.282515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12550.308519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12550.394340] Lustre: lustre-MDT0000: new disk, initializing [12550.522992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12550.575279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12556.072030] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12567.406863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.600614] Lustre: 319966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12567.666281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12567.669225] Lustre: Skipped 1 previous similar message [12567.752993] Lustre: lustre-MDT0001: new disk, initializing [12567.852354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12567.899961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12567.914659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12573.846661] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12586.067653] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12594.002449] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12602.926349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12603.191697] Lustre: lustre-OST0000: new disk, initializing [12603.201765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12603.215857] Lustre: 321933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12603.369649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12610.925466] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12611.192962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12611.203963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12611.361051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12622.911793] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12625.208269] Lustre: Mounted lustre-client - version 2.17.58_42_gf11d80b [12626.303853] Lustre: Unmounted lustre-client [12637.153372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12637.155423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12637.160498] LustreError: Skipped 1 previous similar message [12637.161486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12637.186886] Lustre: Skipped 1 previous similar message [12640.742600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12640.749661] Lustre: Skipped 2 previous similar messages [12643.355728] Lustre: server umount lustre-OST0000 complete [12649.952774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12649.968552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.002350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12650.980922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.764869] Lustre: server umount lustre-MDT0000 complete [12658.444766] LustreError: 319957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982612 with bad export cookie 7799363651733971936 [12658.454252] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12658.456855] LustreError: 319957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12658.789311] Lustre: server umount lustre-MDT0001 complete [12671.907344] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12675.146638] Key type lgssc unregistered [12676.063240] LNet: 1 local NIs in recovery (showing 1): 192.168.202.135@tcp [12676.448523] LNet: 324099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12676.462086] LNetError: 324099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12756.396052] LNet: Removed LNI 192.168.202.135@tcp [12757.488859] Key type .llcrypt unregistered [12757.491053] Key type ._llcrypt unregistered [12778.781553] Key type ._llcrypt registered [12778.786505] Key type .llcrypt registered [12778.919579] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [12799.497721] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12800.930680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12800.965316] alg: No test for adler32 (adler32-zlib) [12802.079646] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12802.332399] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12804.063339] Key type lgssc registered [12805.654644] Lustre: Echo OBD driver; http://www.lustre.org/ [12843.563649] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:39:55 (1788982795) [12853.159635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12853.191496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.637154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12854.679387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12854.791938] Lustre: lustre-MDT0000: new disk, initializing [12854.952294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12854.997643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12860.717865] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12874.251981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12874.398426] Lustre: 327323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12874.448954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12874.453391] Lustre: Skipped 1 previous similar message [12874.529546] Lustre: lustre-MDT0001: new disk, initializing [12874.650939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12874.713823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12874.741819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12880.046513] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12892.554537] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12899.655341] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12908.016689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12908.289982] Lustre: lustre-OST0000: new disk, initializing [12908.296107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12908.302504] Lustre: 329291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12908.423125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12914.276456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12914.292177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12914.433685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12915.314734] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12926.527374] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12945.393774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12945.406046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12945.436185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12945.463229] LustreError: Skipped 1 previous similar message [12949.701761] Lustre: server umount lustre-OST0000 complete [12955.618775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.623895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12955.629616] Lustre: Skipped 2 previous similar messages [12955.640532] Lustre: Skipped 2 previous similar messages [12960.037893] Lustre: server umount lustre-MDT0000 complete [12964.675974] LustreError: 327315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982918 with bad export cookie 13280548362926247788 [12964.680686] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12964.693966] LustreError: 327315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12965.311252] Lustre: server umount lustre-MDT0001 complete [12979.773723] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12982.632347] Key type lgssc unregistered [12983.003869] LNet: 331198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12983.007825] LNetError: 331198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12983.022755] LNet: Removed LNI 192.168.202.135@tcp [12983.934577] Key type .llcrypt unregistered [12983.939464] Key type ._llcrypt unregistered [13006.862603] Key type ._llcrypt registered [13006.864703] Key type .llcrypt registered [13006.961982] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13007.936273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13007.948910] alg: No test for adler32 (adler32-zlib) [13009.361512] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13009.578642] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13011.251992] Key type lgssc registered [13012.553458] Lustre: Echo OBD driver; http://www.lustre.org/ [13021.315808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13021.338375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13022.978290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13027.584855] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13033.459044] LustreError: 332472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13033.478234] LustreError: 332472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13038.179668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13038.617472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13043.501351] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13051.839389] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13059.572700] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13068.277826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13068.613984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13075.467922] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13084.114407] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13107.743916] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13109.805994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13115.361674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13115.368458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13115.384790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13117.167778] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [13118.948747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13118.950826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13118.956721] Lustre: Skipped 1 previous similar message [13118.970453] Lustre: Skipped 1 previous similar message [13120.695694] Lustre: server umount lustre-MDT0000 complete [13122.245615] LustreError: 334575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.070563] LustreError: 332469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.102584] LustreError: 332469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13125.389224] LustreError: 333206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983079 with bad export cookie 13341790059435825698 [13125.397790] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13125.408564] LustreError: 333206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13127.370869] LustreError: 334575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.416205] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [13127.431806] Lustre: Skipped 1 previous similar message [13129.187204] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13132.312685] Lustre: server umount lustre-MDT0001 complete [13137.256706] Lustre: server umount lustre-OST0000 complete [13148.611492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13154.460264] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13158.141755] Lustre: 336675:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.135@tcp [13160.869445] Lustre: server umount MGS complete [13172.587911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.427574] LustreError: 337248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.463369] LustreError: 337248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13173.621633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13179.984683] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13182.947620] LustreError: 337249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13182.998123] LustreError: 337249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13192.642215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13193.188929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13193.208312] Lustre: Skipped 2 previous similar messages [13193.401269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13199.815037] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13211.409829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.030656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13219.906744] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13229.852766] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13231.945406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13239.571819] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13241.728953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13247.975871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13247.984165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13248.017041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13248.994020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13253.700592] Lustre: server umount lustre-OST0000 complete [13259.755342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13259.778194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.799331] Lustre: Skipped 1 previous similar message [13264.355302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13264.365210] Lustre: Skipped 4 previous similar messages [13266.146320] Lustre: server umount lustre-MDT0000 complete [13269.472177] LustreError: 337248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13269.501250] LustreError: 337248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13271.315748] LustreError: 337230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983225 with bad export cookie 13341790059435826545 [13271.318161] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13271.339678] LustreError: 337230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13271.709539] Lustre: server umount lustre-MDT0001 complete [13286.750137] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13290.340544] Key type lgssc unregistered [13290.711309] LNet: 340686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13290.732651] LNetError: 340686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13291.757016] LNet: Removed LNI 192.168.202.135@tcp [13292.847152] Key type .llcrypt unregistered [13292.849697] Key type ._llcrypt unregistered [13316.577658] Key type ._llcrypt registered [13316.581294] Key type .llcrypt registered [13316.754952] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [13336.589757] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13338.627814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13338.685295] alg: No test for adler32 (adler32-zlib) [13339.814977] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13340.153860] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13341.879472] Key type lgssc registered [13343.269525] Lustre: Echo OBD driver; http://www.lustre.org/ [13393.295796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13393.332635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13394.767412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13394.820275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13394.976950] Lustre: lustre-MDT0000: new disk, initializing [13395.134290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13395.153163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13401.650348] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13414.513246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13414.641798] Lustre: 343846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13414.718195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13414.723154] Lustre: Skipped 1 previous similar message [13414.805278] Lustre: lustre-MDT0001: new disk, initializing [13414.933742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13415.009967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13415.026576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13421.080861] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13433.257139] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13441.050440] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13450.129238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13450.572544] Lustre: lustre-OST0000: new disk, initializing [13450.577715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13450.598732] Lustre: 345816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13450.751322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13458.320259] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13460.073278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13460.085400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13460.229235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13469.796143] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13475.589418] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13475.833437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13481.385413] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13481.728594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13486.055518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13486.056120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13486.058917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13486.067036] LustreError: Skipped 1 previous similar message [13486.108460] Lustre: Skipped 1 previous similar message [13491.171111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13491.181900] Lustre: Skipped 2 previous similar messages [13496.292914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13496.305523] Lustre: Skipped 1 previous similar message [13498.335621] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13498.875596] Lustre: server umount lustre-OST0000 complete [13506.532310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13506.567705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13510.901817] Lustre: server umount lustre-MDT0000 complete [13516.393625] LustreError: 343839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983470 with bad export cookie 1198304793190237341 [13516.396817] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13516.411198] LustreError: 343839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13516.821579] Lustre: server umount lustre-MDT0001 complete [13528.420750] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:51:21 (1788983481) [13549.188379] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [13561.672290] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13609.208183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13609.528549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13609.580209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13609.660658] Lustre: lustre-MDT0000: new disk, initializing [13609.733252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13609.766939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13615.840871] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13629.810775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13630.016735] Lustre: 350582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13630.109221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13630.112779] Lustre: Skipped 1 previous similar message [13630.207990] Lustre: lustre-MDT0001: new disk, initializing [13630.332920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13630.385642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13630.403965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13636.310729] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13647.099931] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13653.650131] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13660.622786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13660.860987] Lustre: lustre-OST0000: new disk, initializing [13660.865146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13660.881241] Lustre: 352554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13661.053052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13668.072275] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13671.024903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13671.038414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13671.128304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13678.912811] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13722.592204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13722.595873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13722.610742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13722.613304] Lustre: Skipped 2 previous similar messages [13722.640502] Lustre: Skipped 2 previous similar messages [13728.000127] Lustre: server umount lustre-OST0000 complete [13732.833977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13732.873222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13732.885873] Lustre: Skipped 3 previous similar messages [13737.942362] Lustre: server umount lustre-MDT0000 complete [13737.953643] LustreError: 351665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.953654] LustreError: 351665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13742.630733] LustreError: 351379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983696 with bad export cookie 1198304793190238223 [13742.636067] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13742.639086] LustreError: 351379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13743.352825] Lustre: server umount lustre-MDT0001 complete [13758.356719] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13762.170937] Key type lgssc unregistered [13762.509590] LNet: 354728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13762.526106] LNetError: 354728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13762.547945] LNet: Removed LNI 192.168.202.135@tcp [13763.482539] Key type .llcrypt unregistered [13763.486052] Key type ._llcrypt unregistered [13879.402959] Key type ._llcrypt registered [13879.405665] Key type .llcrypt registered [13879.513685] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13879.806318] Key type .llcrypt unregistered [13879.808113] Key type ._llcrypt unregistered [13978.746432] Key type ._llcrypt registered [13978.749916] Key type .llcrypt registered [13978.916988] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13979.369510] Key type .llcrypt unregistered [13979.374717] Key type ._llcrypt unregistered [14015.294851] Key type ._llcrypt registered [14015.300734] Key type .llcrypt registered [14015.395750] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14015.732378] Key type .llcrypt unregistered [14015.734669] Key type ._llcrypt unregistered [14060.670983] Key type ._llcrypt registered [14060.674321] Key type .llcrypt registered [14060.806065] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14061.084808] Key type .llcrypt unregistered [14061.087451] Key type ._llcrypt unregistered [14155.554514] Key type ._llcrypt registered [14155.557424] Key type .llcrypt registered [14155.677390] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14155.944603] Key type .llcrypt unregistered [14155.948169] Key type ._llcrypt unregistered [14189.184498] Key type ._llcrypt registered [14189.187356] Key type .llcrypt registered [14189.305083] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14189.620320] Key type .llcrypt unregistered [14189.621723] Key type ._llcrypt unregistered [14229.030780] Key type ._llcrypt registered [14229.033675] Key type .llcrypt registered [14229.198740] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14229.674303] Key type .llcrypt unregistered [14229.677375] Key type ._llcrypt unregistered [14265.157663] Key type ._llcrypt registered [14265.161345] Key type .llcrypt registered [14265.337765] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14265.861384] Key type .llcrypt unregistered [14265.880054] Key type ._llcrypt unregistered [14303.081363] Key type ._llcrypt registered [14303.084855] Key type .llcrypt registered [14303.273814] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14303.609452] Key type .llcrypt unregistered [14303.611782] Key type ._llcrypt unregistered [14342.210601] Key type ._llcrypt registered [14342.212073] Key type .llcrypt registered [14342.409752] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14342.832403] Key type .llcrypt unregistered [14342.834201] Key type ._llcrypt unregistered [14415.905328] Key type ._llcrypt registered [14415.907939] Key type .llcrypt registered [14416.063626] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14416.440326] Key type .llcrypt unregistered [14416.442101] Key type ._llcrypt unregistered [14449.463505] Key type ._llcrypt registered [14449.466219] Key type .llcrypt registered [14449.589639] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14450.014397] Key type .llcrypt unregistered [14450.016083] Key type ._llcrypt unregistered [14491.734323] Key type ._llcrypt registered [14491.737308] Key type .llcrypt registered [14491.837066] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14492.128536] Key type .llcrypt unregistered [14492.131757] Key type ._llcrypt unregistered [14556.345570] Key type ._llcrypt registered [14556.348189] Key type .llcrypt registered [14556.447739] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14573.550589] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14574.921816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14575.072086] alg: No test for adler32 (adler32-zlib) [14576.283878] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [14576.511666] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [14578.208888] Key type lgssc registered [14580.099508] Lustre: Echo OBD driver; http://www.lustre.org/ [14624.595825] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14624.679889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14626.093516] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14626.149126] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14626.319433] Lustre: tst250fs-MDT0000: new disk, initializing [14626.467253] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14626.489954] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14632.619572] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14644.722594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14644.908653] Lustre: 375091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14645.007560] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14645.011432] Lustre: Skipped 1 previous similar message [14645.097137] Lustre: tst250fs-MDT0001: new disk, initializing [14645.206663] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14645.235907] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14645.248724] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14652.554284] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14658.548145] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14667.427210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14667.726819] Lustre: tst250fs-OST0000: new disk, initializing [14667.729950] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14667.734255] Lustre: 376727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14667.840893] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14669.501929] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14669.512740] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14669.637382] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14675.518596] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14690.994685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14691.213624] Lustre: tst250fs-OST0001: new disk, initializing [14691.217093] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14691.227438] Lustre: 377599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14691.353292] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14698.184517] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14698.200925] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14698.396398] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14701.076718] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14708.020916] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14710.122320] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14710.125605] Lustre: Skipped 2 previous similar messages [14732.255984] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14732.264520] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14732.287813] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14734.305454] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14734.314431] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14734.330494] Lustre: Skipped 2 previous similar messages [14734.367723] Lustre: Skipped 2 previous similar messages [14739.429947] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14739.441929] Lustre: Skipped 3 previous similar messages [14744.546389] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14744.556401] Lustre: Skipped 3 previous similar messages [14745.055139] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14745.315991] Lustre: server umount tst250fs-MDT0000 complete [14749.461531] LustreError: 376726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788984703 with bad export cookie 17926507746213109376 [14749.462939] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14749.473320] LustreError: 376726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14749.673552] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14749.790422] Lustre: server umount tst250fs-MDT0001 complete [14756.223811] Lustre: server umount tst250fs-OST0000 complete [14760.958834] Lustre: server umount tst250fs-OST0001 complete [14781.127483] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14792.071226] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14831.716569] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:13:04 (1788984784) [14834.210146] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14854.103993] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14866.721994] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14902.213772] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14604 sec ========= 16:14:14 (1788984854) [14904.719604] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:14:17 (1788984857) === [14909.981896] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:14:22 (1788984862) === [14960.444295] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14964.697225] Key type lgssc unregistered [14964.989943] LNet: 386731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14965.012776] LNetError: 386731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14965.062205] LNet: Removed LNI 192.168.202.135@tcp [14966.108280] Key type .llcrypt unregistered [14966.118684] Key type ._llcrypt unregistered