[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522490075 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003332] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006012] kvm-guest: setup PV IPIs [ 0.009378] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010047] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011032] pid_max: default: 32768 minimum: 301 [ 0.012220] LSM: Security Framework initializing [ 0.013100] Yama: becoming mindful. [ 0.014071] SELinux: Initializing. [ 0.016100] *** VALIDATE selinux *** [ 0.024395] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030232] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031206] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034067] *** VALIDATE tmpfs *** [ 0.036673] *** VALIDATE proc *** [ 0.037336] *** VALIDATE cgroup *** [ 0.038012] *** VALIDATE cgroup2 *** [ 0.040274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044031] Spectre V2 : User space: Vulnerable [ 0.045011] Speculative Store Bypass: Vulnerable [ 0.048153] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.050967] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051802] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052027] ... version: 2 [ 0.053017] ... bit width: 48 [ 0.054011] ... generic registers: 4 [ 0.055014] ... value mask: 0000ffffffffffff [ 0.056015] ... max period: 00007fffffffffff [ 0.057015] ... fixed-purpose events: 3 [ 0.058010] ... event mask: 000000070000000f [ 0.059301] rcu: Hierarchical SRCU implementation. [ 0.061806] smp: Bringing up secondary CPUs ... [ 0.062668] x86: Booting SMP configuration: [ 0.063026] .... node #0, CPUs: #1 #2 #3 [ 0.067015] smp: Brought up 1 node, 4 CPUs [ 0.069021] smpboot: Max logical packages: 1 [ 0.070018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.292612] node 0 deferred pages initialised in 219ms [ 0.295175] devtmpfs: initialized [ 0.296280] x86/mm: Memory block size: 128MB [ 0.299048] gcov: version magic: 0x41383552 [ 0.301428] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.306110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.309695] pinctrl core: initialized pinctrl subsystem [ 0.311268] [ 0.311891] ************************************************************* [ 0.314023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.316016] ** ** [ 0.319023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.321017] ** ** [ 0.324019] ** This means that this kernel is built to expose internal ** [ 0.326112] ** IOMMU data structures, which may compromise security on ** [ 0.329017] ** your system. ** [ 0.331019] ** ** [ 0.334020] ** If you see this message and you are not debugging the ** [ 0.336017] ** kernel, report this immediately to your vendor! ** [ 0.339018] ** ** [ 0.341014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.344018] ************************************************************* [ 0.346861] NET: Registered protocol family 16 [ 0.349510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.352067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.355090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.359018] cpuidle: using governor menu [ 0.361031] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.364929] PCI: Using configuration type 1 for base access [ 0.367164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.376326] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.379053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.382183] cryptd: max_cpu_qlen set to 1000 [ 0.385281] ACPI: Added _OSI(Module Device) [ 0.387024] ACPI: Added _OSI(Processor Device) [ 0.389022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.391020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.396193] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.404612] ACPI: Interpreter enabled [ 0.406092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.408020] ACPI: Using IOAPIC for interrupt routing [ 0.410147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.414490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.425308] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.427053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.430028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.434113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.439801] acpiphp: Slot [2] registered [ 0.441202] acpiphp: Slot [5] registered [ 0.443230] acpiphp: Slot [6] registered [ 0.444173] acpiphp: Slot [7] registered [ 0.446159] acpiphp: Slot [8] registered [ 0.447178] acpiphp: Slot [9] registered [ 0.449178] acpiphp: Slot [10] registered [ 0.451190] acpiphp: Slot [3] registered [ 0.452103] acpiphp: Slot [4] registered [ 0.453100] acpiphp: Slot [11] registered [ 0.455124] acpiphp: Slot [12] registered [ 0.456145] acpiphp: Slot [13] registered [ 0.458202] acpiphp: Slot [14] registered [ 0.460132] acpiphp: Slot [15] registered [ 0.461151] acpiphp: Slot [16] registered [ 0.463145] acpiphp: Slot [17] registered [ 0.465147] acpiphp: Slot [18] registered [ 0.467144] acpiphp: Slot [19] registered [ 0.468140] acpiphp: Slot [20] registered [ 0.470191] acpiphp: Slot [21] registered [ 0.472154] acpiphp: Slot [22] registered [ 0.474187] acpiphp: Slot [23] registered [ 0.476163] acpiphp: Slot [24] registered [ 0.477145] acpiphp: Slot [25] registered [ 0.479157] acpiphp: Slot [26] registered [ 0.481185] acpiphp: Slot [27] registered [ 0.483154] acpiphp: Slot [28] registered [ 0.484151] acpiphp: Slot [29] registered [ 0.486151] acpiphp: Slot [30] registered [ 0.488226] acpiphp: Slot [31] registered [ 0.490075] PCI host bridge to bus 0000:00 [ 0.492025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.494035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.496027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.499031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.501028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.504044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.507020] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.511139] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.514284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.523640] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.528675] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.532025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.534023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.537022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.541708] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.545918] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.549054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.553160] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.559020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.573958] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.581022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.587491] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.597021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.607018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.622018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.633482] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.656084] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.685025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.696255] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.702020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.712022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.726019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.738727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.751096] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.764026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.783024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.795774] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.807027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.815031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.839032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.849883] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.864029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.873020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.890021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.902352] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.905472] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.908579] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.911562] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.915289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.922050] iommu: Default domain type: Passthrough [ 0.924556] SCSI subsystem initialized [ 0.926195] ACPI: bus type USB registered [ 0.928112] usbcore: registered new interface driver usbfs [ 0.930086] usbcore: registered new interface driver hub [ 0.932031] usbcore: registered new device driver usb [ 0.933224] pps_core: LinuxPPS API ver. 1 registered [ 0.935013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.939369] PTP clock support registered [ 0.940239] EDAC MC: Ver: 3.0.0 [ 0.942198] PCI: Using ACPI for IRQ routing [ 0.943958] NetLabel: Initializing [ 0.944014] NetLabel: domain hash size = 128 [ 0.945013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.946245] NetLabel: unlabeled traffic allowed by default [ 0.948148] vgaarb: loaded [ 0.949315] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.951012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.957437] clocksource: Switched to clocksource kvm-clock [ 1.070943] VFS: Disk quotas dquot_6.6.0 [ 1.072694] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.075259] *** VALIDATE ramfs *** [ 1.076481] *** VALIDATE hugetlbfs *** [ 1.078088] pnp: PnP ACPI init [ 1.080602] pnp: PnP ACPI: found 6 devices [ 1.100982] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.104324] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.106534] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.108127] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.109637] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.111334] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.114124] NET: Registered protocol family 2 [ 1.116206] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.121065] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.124641] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.129100] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.132245] TCP: Hash tables configured (established 65536 bind 65536) [ 1.134647] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.137237] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.139275] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.141687] NET: Registered protocol family 1 [ 1.144662] RPC: Registered named UNIX socket transport module. [ 1.146323] RPC: Registered udp transport module. [ 1.147468] RPC: Registered tcp transport module. [ 1.148895] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.150852] NET: Registered protocol family 44 [ 1.152426] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.153881] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.156176] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.158587] PCI: CLS 0 bytes, default 64 [ 1.160568] Unpacking initramfs... [ 2.649724] debug: unmapping init [mem 0xffff8c3abcc54000-0xffff8c3abffbffff] [ 2.653663] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.655804] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.658668] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.205836] Initialise system trusted keyrings [ 3.207399] Key type blacklist registered [ 3.210606] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.220227] zbud: loaded [ 3.223273] *** VALIDATE nfs *** [ 3.224489] *** VALIDATE nfs4 *** [ 3.226095] pstore: using deflate compression [ 3.230079] Platform Keyring initialized [ 3.371901] NET: Registered protocol family 38 [ 3.373611] Key type asymmetric registered [ 3.375119] Asymmetric key parser 'x509' registered [ 3.376969] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.379531] io scheduler mq-deadline registered [ 3.380954] io scheduler kyber registered [ 3.382634] io scheduler bfq registered [ 3.385127] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.388421] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.391340] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.394429] ACPI: Power Button [PWRF] [ 3.400302] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.407312] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.430872] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.440751] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.471869] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.511760] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.548223] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.554287] Non-volatile memory driver v1.3 [ 3.556234] Linux agpgart interface v0.103 [ 3.598382] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.603658] vda: detected capacity change from 0 to 75071488 [ 3.622607] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.626150] vdb: detected capacity change from 0 to 1073741824 [ 3.643776] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.648376] vdc: detected capacity change from 0 to 2621440000 [ 3.667714] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.671046] vdd: detected capacity change from 0 to 2621440000 [ 3.689651] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.693062] vde: detected capacity change from 0 to 4294967296 [ 3.712615] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.715911] vdf: detected capacity change from 0 to 4294967296 [ 3.727194] libphy: Fixed MDIO Bus: probed [ 3.735471] usbcore: registered new interface driver usbserial_generic [ 3.738358] usbserial: USB Serial support registered for generic [ 3.740936] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.745524] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.747702] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.751140] mousedev: PS/2 mouse device common for all mice [ 3.755209] rtc_cmos 00:05: RTC can wake from S4 [ 3.758775] rtc_cmos 00:05: registered as rtc0 [ 3.760914] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.764066] intel_pstate: CPU model not supported [ 3.767768] hid: raw HID events driver (C) Jiri Kosina [ 3.769810] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.773120] usbcore: registered new interface driver usbhid [ 3.773130] usbhid: USB HID core driver [ 3.773290] drop_monitor: Initializing network drop monitor service [ 3.773467] Initializing XFRM netlink socket [ 3.773834] NET: Registered protocol family 10 [ 3.775362] Segment Routing with IPv6 [ 3.782496] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.783098] NET: Registered protocol family 17 [ 3.787826] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.789369] mpls_gso: MPLS GSO support [ 3.803437] RAS: Correctable Errors collector initialized. [ 3.805972] AVX version of gcm_enc/dec engaged. [ 3.807548] AES CTR mode by8 optimization enabled [ 3.901784] sched_clock: Marking stable (3901753873, 0)->(4880921264, -979167391) [ 3.906477] registered taskstats version 1 [ 3.908406] Loading compiled-in X.509 certificates [ 3.912362] zswap: loaded using pool lzo/zbud [ 3.943729] Key type big_key registered [ 3.956123] Key type encrypted registered [ 3.957842] ima: No TPM chip found, activating TPM-bypass! [ 3.960183] ima: Allocated hash algorithm: sha1 [ 3.962095] ima: No architecture policies found [ 3.964159] evm: Initialising EVM extended attributes: [ 3.966285] evm: security.selinux [ 3.967693] evm: security.ima [ 3.969382] evm: security.capability [ 3.970747] evm: HMAC attrs: 0x1 [ 3.973399] rtc_cmos 00:05: setting system clock to 2026-08-27 01:04:59 UTC (1787792699) [ 3.986772] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.989734] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 4.005117] Write protecting the kernel read-only data: 28672k [ 4.009773] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 4.012954] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 4.078820] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.087435] systemd[1]: Detected virtualization kvm. [ 4.089602] systemd[1]: Detected architecture x86-64. [ 4.091630] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.124086] systemd[1]: No hostname configured. [ 4.126100] systemd[1]: Set hostname to . [ 4.128610] random: systemd: uninitialized urandom read (16 bytes read) [ 4.131851] systemd[1]: Initializing machine ID from random generator. [ 4.306633] random: systemd: uninitialized urandom read (16 bytes read) [ 4.310000] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.315976] random: systemd: uninitialized urandom read (16 bytes read) [ 4.320103] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.328405] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.052059] device-mapper: uevent: version 1.0.3 [ 5.054770] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.935575] virtio_net virtio0 ens2: renamed from eth0 [ 5.967609] random: fast init done [ 6.163824] scsi host0: ata_piix [ 6.187704] scsi host1: ata_piix [ 6.189869] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.191940] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.415237] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.477623] random: crng init done [ 10.479046] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.270306] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.879421] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.272285] SELinux: Disabled at runtime. [ 13.339382] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.353474] systemd[1]: Detected virtualization kvm. [ 13.356473] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.035498] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.041218] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.047757] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.053360] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.057826] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.068781] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.078179] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ 14.131169] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.685417] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.108389] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.156317] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.295272] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.314730] EDAC sbridge: Ver: 1.1.2 [ 17.722372] Key type dns_resolver registered [ 18.044990] NFS: Registering the id_resolver key type [ 18.047177] Key type id_resolver registered [ 18.048629] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg438-server login: [ 64.367969] libcfs: loading out-of-tree module taints kernel. [ 64.446164] Key type ._llcrypt registered [ 64.448552] Key type .llcrypt registered [ 64.599775] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 65.047467] hrtimer: interrupt took 5459412 ns [ 89.475429] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 91.380061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.405804] alg: No test for adler32 (adler32-zlib) [ 93.067424] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 94.088361] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 95.896323] Key type lgssc registered [ 98.030391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.963643] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.238563] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 165.973047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.057792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.342451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.377379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.471269] Lustre: lustre-MDT0000: new disk, initializing [ 167.577527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.597931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.089353] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.544104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.632568] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.657684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.665234] Lustre: Skipped 1 previous similar message [ 190.855223] Lustre: lustre-MDT0001: new disk, initializing [ 191.004342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.081956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 191.111625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.091970] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.000667] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.922924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.444869] Lustre: lustre-OST0000: new disk, initializing [ 214.464107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.477586] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.594872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.305452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.315153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.353603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.155271] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.286345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.496317] Lustre: lustre-OST0001: new disk, initializing [ 237.501364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.508257] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.589675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.813846] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.316810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.330121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.499518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 257.834982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.930958] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.979917] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.068457] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing yml_node [ 284.220507] Lustre: DEBUG MARKER: Client: 2.17.57.57 [ 286.457449] Lustre: DEBUG MARKER: MDS: 2.17.57.57 [ 288.524386] Lustre: DEBUG MARKER: OSS: 2.17.57.57 [ 289.907704] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 21:09:43 EDT 2026 [ 306.342142] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 308.116337] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 327.651636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.653351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.663865] Lustre: Skipped 1 previous similar message [ 330.889897] Lustre: server umount lustre-MDT0000 complete [ 334.484237] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793030 with bad export cookie 16615272088509839731 [ 334.486604] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.495929] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 337.889398] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.890604] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.891454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.891459] Lustre: Skipped 3 previous similar messages [ 337.904371] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 337.937068] Lustre: Skipped 3 previous similar messages [ 339.937136] LustreError: 9375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.937165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.976072] Lustre: Skipped 2 previous similar messages [ 340.860888] Lustre: server umount lustre-MDT0001 complete [ 344.968381] Lustre: server umount lustre-OST0000 complete [ 348.862048] Lustre: server umount lustre-OST0001 complete [ 355.908652] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 365.284711] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 400.680852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.945821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.983517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.098854] Lustre: lustre-MDT0000: new disk, initializing [ 401.241103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.290538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.128956] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 417.399955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.544057] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.604880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.608692] Lustre: Skipped 1 previous similar message [ 417.685755] Lustre: lustre-MDT0001: new disk, initializing [ 417.784295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.867012] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.881691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.842714] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 433.268562] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.055289] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.399545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.850844] Lustre: lustre-OST0000: new disk, initializing [ 447.861142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.894261] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.089587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.631240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.654285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.931789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.195576] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 467.354852] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.840179] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.087450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.452651] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.632355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.738707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.751472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.752800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.774329] Lustre: Skipped 1 previous similar message [ 485.861422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.869756] Lustre: Skipped 1 previous similar message [ 486.665418] Lustre: server umount lustre-OST0000 complete [ 494.563873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 494.578373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.590627] Lustre: Skipped 1 previous similar message [ 494.599324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.512667] Lustre: server umount lustre-MDT0000 complete [ 501.221406] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.235935] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 501.464948] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793196 with bad export cookie 16615272088509841271 [ 501.470458] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.480929] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 501.944554] Lustre: server umount lustre-MDT0001 complete [ 507.817163] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:13:20 (1787793200) [ 526.883497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.341271] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.406058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.031381] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 532.452889] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.929055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.400879] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 554.561910] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 560.485479] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.384356] Lustre: Failing over lustre-MDT0000 [ 562.731874] Lustre: server umount lustre-MDT0000 complete [ 566.517175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.523900] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.530822] Lustre: Skipped 2 previous similar messages [ 566.559564] LustreError: 19920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 582.688541] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793262/real 1787793262] req@ffff8c3b3d7a8b40 x1874636611681024/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787793278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 582.731774] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.868958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.849634] LustreError: 20647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.863040] LustreError: 20647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 592.997156] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe6954deeda1784fd [ 593.016694] Lustre: MGC192.168.204.138@tcp: Connection restored to 0@lo (at 0@lo) [ 593.256347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.266455] Lustre: Skipped 1 previous similar message [ 598.049594] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 598.507758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 598.518408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 598.546036] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 607.654982] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 612.583383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.593480] Lustre: Skipped 1 previous similar message [ 612.598644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.607464] Lustre: Skipped 3 previous similar messages [ 617.963783] Lustre: server umount lustre-MDT0000 complete [ 621.942238] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793317 with bad export cookie 16615272088509842685 [ 621.956478] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.468904] Lustre: server umount lustre-MDT0001 complete [ 631.284697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.712954] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.734370] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 635.983278] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 644.602966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.196607] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 656.705622] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 662.855929] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.288470] Lustre: Failing over lustre-MDT0000 [ 665.571830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 665.584626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.669877] Lustre: server umount lustre-MDT0000 complete [ 683.938054] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793363/real 1787793363] req@ffff8c3b3d7aa580 x1874636611710208/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787793379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 683.969576] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 684.832119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.582474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 699.035920] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 699.874540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 699.899449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 699.908070] Lustre: Skipped 1 previous similar message [ 699.945618] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 709.371412] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 714.212082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.217212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.231471] Lustre: Skipped 2 previous similar messages [ 714.253182] Lustre: Skipped 1 previous similar message [ 719.678812] Lustre: server umount lustre-MDT0000 complete [ 720.358723] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.376501] LustreError: 23256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 723.063168] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793418 with bad export cookie 16615272088509843623 [ 723.067625] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.080247] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 723.477589] Lustre: server umount lustre-MDT0001 complete [ 731.384591] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:17:04 (1787793424) [ 747.421342] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 758.232990] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 801.632822] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 812.259813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.559880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 812.592377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 812.731412] Lustre: lustre-MDT0000: new disk, initializing [ 812.886361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.903470] Lustre: Skipped 3 previous similar messages [ 812.924154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 818.340599] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 829.496544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.677849] Lustre: 29634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 830.048378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 834.477974] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 843.886360] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 849.707897] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 857.160454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.468060] Lustre: 31600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 859.383753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 859.439135] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 859.527989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 859.542587] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 864.446529] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 875.035289] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.405764] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 905.706510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.713835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.728543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.761753] Lustre: Skipped 1 previous similar message [ 905.798363] Lustre: Skipped 3 previous similar messages [ 908.361236] Lustre: server umount lustre-OST0000 complete [ 927.200183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 927.413353] Lustre: server umount lustre-MDT0000 complete [ 931.135466] LustreError: 30427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793626 with bad export cookie 16615272088509844022 [ 931.140434] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.164130] LustreError: 30427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 931.738401] Lustre: server umount lustre-MDT0001 complete [ 943.215739] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 946.331335] Key type lgssc unregistered [ 946.656852] LNet: 33586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.673378] LNetError: 33586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.698914] LNet: Removed LNI 192.168.204.138@tcp [ 947.843159] Key type .llcrypt unregistered [ 947.845395] Key type ._llcrypt unregistered [ 971.662969] Key type ._llcrypt registered [ 971.665081] Key type .llcrypt registered [ 971.793560] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 972.940177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.984559] alg: No test for adler32 (adler32-zlib) [ 974.064598] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 974.359127] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 976.051138] Key type lgssc registered [ 977.370517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.421335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.444108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.995306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.731500] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 997.347703] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.393748] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1001.049945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.536549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.940429] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1013.658627] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1019.212963] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1027.063655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1027.641672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.824629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1034.976718] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1042.893124] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1046.156045] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1060.676724] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:22:34 (1787793754) [ 1062.519240] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1064.266769] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1070.821828] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:22:44 (1787793764) [ 1078.775226] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:22:52 (1787793772) [ 1079.812108] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1088.114819] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1088.118619] Lustre: Skipped 2 previous similar messages [ 1089.768261] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:23:03 (1787793783) [ 1093.435052] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1093.442688] Lustre: Skipped 3 previous similar messages [ 1116.302025] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1116.310686] Lustre: Skipped 6 previous similar messages [ 1120.915978] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:23:33 (1787793813) [ 1124.292360] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1128.107078] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1137.558492] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:23:50 (1787793830) [ 1140.013050] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1140.025796] Lustre: Skipped 3 previous similar messages [ 1155.922388] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:24:08 (1787793848) [ 1170.401743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.415803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.435985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.499179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.503240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1174.524875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.375085] Lustre: server umount lustre-OST0000 complete [ 1180.641621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1180.651142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.663267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.669347] Lustre: Skipped 1 previous similar message [ 1184.738099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.741368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.604786] Lustre: server umount lustre-MDT0000 complete [ 1189.861410] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.896713] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1190.083170] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793885 with bad export cookie 14598607498165634129 [ 1190.088540] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.098156] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1190.474568] Lustre: server umount lustre-MDT0001 complete [ 1200.359379] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 1202.821914] Key type lgssc unregistered [ 1203.010220] LNet: 42857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.016788] LNetError: 42857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.031626] LNet: Removed LNI 192.168.204.138@tcp [ 1203.938938] Key type .llcrypt unregistered [ 1203.941704] Key type ._llcrypt unregistered [ 1225.652895] Key type ._llcrypt registered [ 1225.658438] Key type .llcrypt registered [ 1225.814777] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 1241.559153] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 1242.488188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.526045] alg: No test for adler32 (adler32-zlib) [ 1243.730120] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1244.021071] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 1245.673116] Key type lgssc registered [ 1246.751670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1278.345759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1278.384306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.602179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1279.624557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1279.775644] Lustre: lustre-MDT0000: new disk, initializing [ 1279.889142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.907689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1283.887427] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1295.158322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.228982] Lustre: 46011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1295.269825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1295.273781] Lustre: Skipped 1 previous similar message [ 1295.332778] Lustre: lustre-MDT0001: new disk, initializing [ 1295.423718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1295.466157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1295.484056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1299.771450] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1307.758320] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1313.451618] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1319.903427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1320.173285] Lustre: lustre-OST0000: new disk, initializing [ 1320.178431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1320.186450] Lustre: 47984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1320.278782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.451881] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1330.267753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1330.281912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1330.442881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1335.861159] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1340.575860] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1340.796215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1345.252918] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1345.464787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1351.143729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.145825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1351.153926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.173772] Lustre: Skipped 1 previous similar message [ 1351.187741] LustreError: Skipped 1 previous similar message [ 1353.605732] Lustre: server umount lustre-OST0000 complete [ 1361.383954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.386696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.420661] Lustre: Skipped 1 previous similar message [ 1361.453517] Lustre: Skipped 2 previous similar messages [ 1363.571651] Lustre: server umount lustre-MDT0000 complete [ 1367.072140] LustreError: 46818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794062 with bad export cookie 12129710963002615825 [ 1367.072907] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.087904] LustreError: 46818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1367.580513] Lustre: server umount lustre-MDT0001 complete [ 1377.417246] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:27:51 (1787794071) [ 1383.805796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.243382] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.254923] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1384.327257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.148464] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1389.546687] LustreError: 50444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.636440] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.217060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.551890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.956874] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1410.400780] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1416.460419] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1423.723412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.146379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.223703] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1438.352966] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1446.072797] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1446.944550] Lustre: Modifying parameter general.timeout=1 in log params [ 1448.688731] Lustre: Modifying parameter general.timeout=3 in log params [ 1448.691875] Lustre: Skipped 1 previous similar message [ 1451.315988] Lustre: Modifying parameter general.timeout=6 in log params [ 1451.321991] Lustre: Skipped 2 previous similar messages [ 1456.023777] Lustre: Modifying parameter general.timeout=11 in log params [ 1456.030433] Lustre: Skipped 4 previous similar messages [ 1464.028754] Lustre: Modifying parameter general.timeout=19 in log params [ 1464.032610] Lustre: Skipped 7 previous similar messages [ 1480.555402] Lustre: Modifying parameter general.timeout=37 in log params [ 1480.564722] Lustre: Skipped 17 previous similar messages [ 1512.698534] Lustre: Modifying parameter general.timeout=71 in log params [ 1512.703089] Lustre: Skipped 33 previous similar messages [ 1568.752443] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:31:02 (1787794262) [ 1571.461256] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1573.197937] Lustre: 60022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1576.658908] Lustre: 60220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1576.669844] Lustre: 60220:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1577.508783] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1577.514890] Lustre: Skipped 64 previous similar messages [ 1604.550907] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1604.563118] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1604.600490] LustreError: 61801:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1607.923643] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:31:41 (1787794301) [ 1618.401782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.408702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.412448] Lustre: Skipped 1 previous similar message [ 1618.426576] Lustre: Skipped 2 previous similar messages [ 1623.529404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.537629] Lustre: Skipped 2 previous similar messages [ 1628.655039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.668042] Lustre: Skipped 2 previous similar messages [ 1632.224415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1632.482616] Lustre: server umount lustre-MDT0000 complete [ 1633.762529] LustreError: 51463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.790627] LustreError: 51463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1636.305973] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794331 with bad export cookie 12129710963002616700 [ 1636.319852] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.322854] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1636.695130] Lustre: server umount lustre-MDT0001 complete [ 1651.079253] Lustre: server umount lustre-OST0000 complete [ 1681.535325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.889866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1682.190546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.608639] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1695.346473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.533407] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1695.575575] Lustre: 64332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1700.645907] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1707.530696] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.093034] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.350468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.547575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1720.844454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.853698] Lustre: Skipped 1 previous similar message [ 1727.266904] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1734.662492] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1736.174090] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1736.187717] Lustre: Skipped 17 previous similar messages [ 1736.266054] Lustre: 66900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1744.868422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.869102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.893117] Lustre: Skipped 1 previous similar message [ 1744.894446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.905612] LustreError: Skipped 1 previous similar message [ 1749.384798] Lustre: server umount lustre-OST0000 complete [ 1755.105627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.126465] Lustre: Skipped 2 previous similar messages [ 1765.349981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.363289] Lustre: Skipped 6 previous similar messages [ 1767.393428] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.770715] Lustre: server umount lustre-MDT0000 complete [ 1770.466715] LustreError: 64338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1770.490633] LustreError: 64338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1771.425742] LustreError: 64325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794466 with bad export cookie 12129710963002619731 [ 1771.429991] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.444068] LustreError: 64325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1771.795516] Lustre: server umount lustre-MDT0001 complete [ 1784.013467] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 1787.269114] Key type lgssc unregistered [ 1787.648732] LNet: 68116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.656686] LNetError: 68116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.672775] LNet: Removed LNI 192.168.204.138@tcp [ 1788.370941] Key type .llcrypt unregistered [ 1788.373085] Key type ._llcrypt unregistered [ 1818.483466] Key type ._llcrypt registered [ 1818.498994] Key type .llcrypt registered [ 1818.644719] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 1820.068092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.132711] alg: No test for adler32 (adler32-zlib) [ 1821.311646] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 1821.588130] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 1823.296445] Key type lgssc registered [ 1824.557701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.782493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1832.803910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.403484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.984516] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1843.683630] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.716143] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1848.825142] LustreError: 69531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.895468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.669920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1856.041967] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1864.895341] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1870.945473] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1878.722578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.721716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.536907] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1896.059116] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1914.242743] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1925.091334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.092852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.113704] Lustre: Skipped 2 previous similar messages [ 1925.116794] Lustre: Skipped 1 previous similar message [ 1930.211170] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.232370] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1931.746547] Lustre: server umount lustre-MDT0000 complete [ 1935.197349] LustreError: 69512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794630 with bad export cookie 8962507081013862917 [ 1935.203402] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.213757] LustreError: 69512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1935.329400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.332953] LustreError: 69526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.342797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1935.342805] Lustre: Skipped 1 previous similar message [ 1935.507917] Lustre: server umount lustre-MDT0001 complete [ 1942.161783] Lustre: server umount lustre-OST0000 complete [ 1972.298824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.628785] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1972.659675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1972.911402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.724538] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 1989.228620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.357680] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1989.395818] Lustre: 74430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.974324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1994.493786] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2003.033935] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2009.068231] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2016.969540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.283500] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2017.793407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.052587] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2033.156710] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2034.658226] Lustre: 77004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2036.467249] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2043.875402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.880062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.896521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.461226] Lustre: server umount lustre-OST0000 complete [ 2054.628073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.646068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.653598] Lustre: Skipped 2 previous similar messages [ 2054.656273] Lustre: Skipped 2 previous similar messages [ 2057.489920] Lustre: server umount lustre-MDT0000 complete [ 2059.748124] LustreError: 74441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.775558] LustreError: 74441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2061.549779] LustreError: 74420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794757 with bad export cookie 8962507081013863764 [ 2061.558994] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.565344] LustreError: 74420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2062.020968] Lustre: server umount lustre-MDT0001 complete [ 2074.254906] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 2077.534873] Key type lgssc unregistered [ 2077.852753] LNet: 78216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.859344] LNetError: 78216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.877262] LNet: Removed LNI 192.168.204.138@tcp [ 2078.823262] Key type .llcrypt unregistered [ 2078.825820] Key type ._llcrypt unregistered [ 2102.261515] Key type ._llcrypt registered [ 2102.263690] Key type .llcrypt registered [ 2102.419709] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 2103.594428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.803577] alg: No test for adler32 (adler32-zlib) [ 2104.879769] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2105.142181] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 2106.864215] Key type lgssc registered [ 2108.726951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2116.549046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2116.577239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.080213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.574332] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2128.357860] LustreError: 79630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.390456] LustreError: 79630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2130.695075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.266482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.357176] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2143.558277] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2149.397987] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2155.378368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.940136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.058238] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2169.582617] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.651080] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.532326] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.243064] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.263657] Lustre: Skipped 1 previous similar message [ 2180.842754] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2180.850531] Lustre: Skipped 2 previous similar messages [ 2184.991308] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2185.000648] Lustre: Skipped 4 previous similar messages [ 2193.292333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2193.302207] Lustre: Skipped 11 previous similar messages [ 2209.360512] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2209.366989] Lustre: Skipped 18 previous similar messages [ 2241.441611] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2241.447275] Lustre: Skipped 39 previous similar messages [ 2263.161146] Lustre: 87726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2265.675473] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:42:39 (1787794959) [ 2267.149213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2269.025251] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:42:42 (1787794962) [ 2274.787751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2274.805108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.835795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.809826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.812381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.812994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.856735] Lustre: Skipped 1 previous similar message [ 2279.443615] Lustre: server umount lustre-OST0000 complete [ 2284.513888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.525876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.545590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.170668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.183219] Lustre: Skipped 3 previous similar messages [ 2296.290612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.314198] Lustre: Skipped 1 previous similar message [ 2298.336528] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2298.539839] Lustre: server umount lustre-MDT0000 complete [ 2301.412208] LustreError: 79626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.437949] LustreError: 79626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2302.892817] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794998 with bad export cookie 6340063882521382269 [ 2302.904733] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.908837] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.357452] Lustre: server umount lustre-MDT0001 complete [ 2313.880956] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 2316.640880] Key type lgssc unregistered [ 2316.872541] LNet: 88980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.884080] LNetError: 88980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.899769] LNet: Removed LNI 192.168.204.138@tcp [ 2317.752168] Key type .llcrypt unregistered [ 2317.758972] Key type ._llcrypt unregistered [ 2323.024336] Key type ._llcrypt registered [ 2323.026243] Key type .llcrypt registered [ 2323.137532] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2327.718921] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules [ 2328.680451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.698529] alg: No test for adler32 (adler32-zlib) [ 2329.868505] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2330.233896] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 2331.969101] Key type lgssc registered [ 2333.284702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.651976] LustreError: 90185:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2351.873566] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 2356.309329] LustreError: 90185:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2361.657371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2361.674793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.095491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.115671] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2372.587950] LustreError: 91208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.619045] LustreError: 91208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2377.301529] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:44:30 (1787795070) [ 2377.699796] LustreError: 91209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.532362] LustreError: 91240:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2381.545135] LustreError: 91240:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2381.551284] LustreError: 91240:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2381.898965] Lustre: server umount lustre-MDT0000 complete [ 2395.478564] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 2398.692641] Key type lgssc unregistered [ 2399.017843] LNet: 92735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.029486] LNetError: 92735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.117875] LNet: Removed LNI 192.168.204.138@tcp [ 2400.910427] Key type .llcrypt unregistered [ 2400.913968] Key type ._llcrypt unregistered [ 2418.153825] Key type ._llcrypt registered [ 2418.155913] Key type .llcrypt registered [ 2418.247209] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 2419.524880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.558325] alg: No test for adler32 (adler32-zlib) [ 2420.613094] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 2420.882736] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 2422.552264] Key type lgssc registered [ 2423.689240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2431.470636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2431.489673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.027977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.579963] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2443.247168] LustreError: 93954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.262782] LustreError: 93954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2445.608605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.213114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.128922] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2456.857749] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2462.954461] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2470.440615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.382309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.016889] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2490.578779] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2538.600079] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:47:12 (1787795232) [ 2540.267680] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2542.218228] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:47:15 (1787795235) [ 2548.197275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2548.202149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.216677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.221542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.223879] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.321873] Lustre: server umount lustre-MDT0000 complete [ 2554.343755] LustreError: 95803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.358982] LustreError: 95803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2556.431468] LustreError: 93933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795251 with bad export cookie 18337430935132463463 [ 2556.439332] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.447137] LustreError: 93933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2556.707058] Lustre: server umount lustre-MDT0001 complete [ 2570.599938] Lustre: server umount lustre-OST0000 complete [ 2580.063682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.489416] LustreError: 98251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.698880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.680711] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2586.087388] LustreError: 98252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.204927] LustreError: 98251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.922940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.658391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.171413] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2605.214187] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2611.209628] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2625.406674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.523195] LustreError: 100341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.654426] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2625.661816] LustreError: 98241:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2625.684468] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2625.693246] LustreError: 100335:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2625.700710] LustreError: 100335:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2625.704737] LustreError: 100335:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2625.775600] Lustre: server umount lustre-OST0000 complete [ 2625.780983] LustreError: 100335:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2633.567616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.797860] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2633.803900] LustreError: 98239:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2633.835722] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2633.844808] LustreError: 100800:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2633.850504] LustreError: 100800:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2633.855675] LustreError: 100800:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2633.926310] Lustre: server umount lustre-OST0000 complete [ 2633.931343] LustreError: 100800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2643.997713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.332570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2644.337111] Lustre: Found index 0 for lustre-OST0000, updating log [ 2644.342890] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2644.474238] Lustre: lustre-OST0000: new disk, initializing [ 2644.478821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2644.487947] Lustre: 101338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2644.866057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.137694] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2654.962958] Lustre: Failing over lustre-OST0000 [ 2655.189598] Lustre: server umount lustre-OST0000 complete [ 2659.809469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2659.821206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.845827] Lustre: Skipped 1 previous similar message [ 2659.851092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.856916] Lustre: Skipped 2 previous similar messages [ 2663.396325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.414315] Lustre: Skipped 3 previous similar messages [ 2665.643729] Lustre: server umount lustre-MDT0000 complete [ 2668.514911] LustreError: 99724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.538336] LustreError: 99724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2669.740975] LustreError: 98233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795365 with bad export cookie 18337430935132466207 [ 2669.744056] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.762479] LustreError: 98233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2670.068852] Lustre: server umount lustre-MDT0001 complete [ 2678.599157] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:49:31 (1787795371) [ 2695.770254] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 2705.073989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.432735] LustreError: 103681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.541200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.957439] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2718.281078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.713831] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2726.691282] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2732.706541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.617805] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2747.953466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.044607] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2748.110301] Lustre: lustre-OST0001: new disk, initializing [ 2748.113367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2748.118115] Lustre: 105999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2748.419146] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.423953] Lustre: Skipped 2 previous similar messages [ 2754.457229] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2754.673808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2754.684512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2754.773107] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2762.188659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2766.833055] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2772.218289] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.38@tcp (stopping) [ 2775.527410] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.539583] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.554915] Lustre: Skipped 2 previous similar messages [ 2780.646451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2780.652352] Lustre: Skipped 4 previous similar messages [ 2785.248179] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2785.474892] Lustre: server umount lustre-MDT0001 complete [ 2785.766212] LustreError: 103676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.784743] LustreError: 103676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2794.891905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.981906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2794.986668] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2795.017288] Lustre: 103671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2796.283144] LustreError: 107551:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2796.304648] LustreError: 107551:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2796.322788] LustreError: 103674:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2796.452563] LustreError: 107571:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.138@tcp: cfg command failed: rc = -17 [ 2796.473562] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2796.473562] [ 2799.780485] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 2800.629299] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2808.450953] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:51:41 (1787795501) [ 2825.575564] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 2844.105983] Lustre: 109615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2869.507673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2873.880736] Lustre: 110751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2964.449854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.454780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.457496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.457505] Lustre: Skipped 3 previous similar messages [ 2964.527551] Lustre: Skipped 3 previous similar messages [ 2967.521591] Lustre: server umount lustre-MDT0000 complete [ 2969.575604] LustreError: 103677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.602486] LustreError: 103677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2972.060090] LustreError: 106719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795667 with bad export cookie 18337430935132467208 [ 2972.065564] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.074314] LustreError: 106719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2972.506928] Lustre: server umount lustre-MDT0001 complete [ 2988.607908] Lustre: server umount lustre-OST0000 complete [ 2990.881674] Lustre: 93265:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795670/real 1787795670] req@ffff8c3b089c1a40 x1874639051912448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2990.920942] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.181183] Lustre: server umount lustre-OST0001 complete [ 3000.363689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3010.546327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3019.643425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3030.503348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3049.270430] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 3059.150909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.171795] Lustre: lustre-MDT0000: reset Object Index mappings [ 3059.719841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.726409] Lustre: Skipped 1 previous similar message [ 3064.121693] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3073.028094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.049225] Lustre: lustre-MDT0001: reset Object Index mappings [ 3078.764713] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3082.250536] Lustre: 115855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3090.447763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.861076] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3100.073946] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3105.306815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3111.054707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.054446] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3117.064395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3119.172910] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3127.316273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3130.946632] Lustre: 117733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3188.051459] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:58:01 (1787795881) [ 3214.314153] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.330715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.344163] Lustre: Skipped 2 previous similar messages [ 3214.346803] Lustre: Skipped 4 previous similar messages [ 3219.593794] Lustre: server umount lustre-MDT0000 complete [ 3223.840344] LustreError: 114687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795919 with bad export cookie 18337430935132514038 [ 3223.844443] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.866327] LustreError: 114687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3228.591099] LustreError: 116233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.628970] LustreError: 116233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3238.369351] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.838855] Lustre: server umount lustre-MDT0001 complete [ 3257.312829] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3257.526489] Lustre: server umount lustre-OST0000 complete [ 3267.699701] Lustre: server umount lustre-OST0001 complete [ 3275.516774] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 3287.191470] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 3326.649136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.851758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3326.889917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3326.987969] Lustre: lustre-MDT0000: new disk, initializing [ 3327.092962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.098961] Lustre: Skipped 3 previous similar messages [ 3327.116855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3332.132172] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3339.698631] Lustre: server umount lustre-MDT0000 complete [ 3350.003729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.337684] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3350.349615] Lustre: 122003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3350.368259] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3355.429422] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3361.769940] Lustre: server umount lustre-MDT0000 complete [ 3371.734241] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:01:04 (1787796064) [ 3373.795552] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3375.759459] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:01:09 (1787796069) [ 3394.770330] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 3405.824915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.508638] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3421.333210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.506932] Lustre: 124064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3421.577449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3421.584638] Lustre: Skipped 1 previous similar message [ 3421.746630] Lustre: lustre-MDT0001: new disk, initializing [ 3421.915797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3421.928712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3427.234759] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3432.514231] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3439.895677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.226722] Lustre: lustre-OST0000: new disk, initializing [ 3440.231533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3440.238607] Lustre: 125645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3441.802156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3441.815084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3441.924997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3446.930527] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3458.318438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.515198] Lustre: lustre-OST0001: new disk, initializing [ 3458.518951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3458.527848] Lustre: 126514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3459.966069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3459.974621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3460.087390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3465.497676] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 3476.290406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3481.223933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3880.195592] Lustre: 132932:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3880.213476] Lustre: 132932:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3880.220408] Lustre: 132932:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3880.234829] Lustre: 132932:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3880.249035] Lustre: 132932:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3880.265476] Lustre: 132932:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4180.510870] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:14:32 (1787796872) [ 4200.425389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.438602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4200.445276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.445288] Lustre: Skipped 12 previous similar messages [ 4200.451651] Lustre: Skipped 5 previous similar messages [ 4202.536977] Lustre: server umount lustre-MDT0000 complete [ 4205.559702] LustreError: 127322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.588190] LustreError: 127322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4206.512209] LustreError: 124059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796902 with bad export cookie 18337430935132582946 [ 4206.519317] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4206.522990] LustreError: 124059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4210.657883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.668723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4210.681152] Lustre: Skipped 2 previous similar messages [ 4210.696055] Lustre: Skipped 4 previous similar messages [ 4213.183482] Lustre: server umount lustre-MDT0001 complete [ 4223.601808] Lustre: server umount lustre-OST0000 complete [ 4234.606641] Lustre: server umount lustre-OST0001 complete [ 4242.782597] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 4254.947910] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 4296.416269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.719666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4296.780191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4296.895321] Lustre: lustre-MDT0000: new disk, initializing [ 4297.015647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.037961] Lustre: Skipped 5 previous similar messages [ 4297.059407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4302.190763] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 4314.057947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.194084] Lustre: 135986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4314.247171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4314.255285] Lustre: Skipped 1 previous similar message [ 4314.358871] Lustre: lustre-MDT0001: new disk, initializing [ 4314.570489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4314.584714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4319.469082] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 4329.798122] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4336.478241] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4344.236537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.680702] Lustre: lustre-OST0000: new disk, initializing [ 4344.709199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4344.723238] Lustre: 137957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4344.837918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.848951] Lustre: Skipped 1 previous similar message [ 4346.690160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4346.702038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4346.865573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4352.178337] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 4362.479176] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.609149] Lustre: lustre-MDD0000: changelog on [ 4375.363088] Lustre: lustre-MDD0001: changelog on [ 6171.265452] Lustre: lustre-MDD0001: changelog off [ 6173.345988] Lustre: 139580:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6174.406436] Lustre: lustre-MDD0000: changelog off [ 6184.928967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6184.936402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.948836] LustreError: Skipped 1 previous similar message [ 6184.973180] Lustre: Skipped 1 previous similar message [ 6184.973830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6184.996288] Lustre: Skipped 3 previous similar messages [ 6187.535212] Lustre: server umount lustre-OST0000 complete [ 6193.645290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6193.653113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.678274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.124882] Lustre: server umount lustre-MDT0000 complete [ 6200.315762] LustreError: 135999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.350927] LustreError: 135999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6205.009453] LustreError: 135978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798900 with bad export cookie 18337430935133404326 [ 6205.021901] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.022835] LustreError: 135978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6205.820947] Lustre: server umount lustre-MDT0001 complete [ 6219.680416] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 6222.841889] Key type lgssc unregistered [ 6223.211225] LNet: 147082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.218666] LNetError: 147082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.236768] LNet: Removed LNI 192.168.204.138@tcp [ 6224.359293] Key type .llcrypt unregistered [ 6224.363068] Key type ._llcrypt unregistered [ 6246.420472] Key type ._llcrypt registered [ 6246.426115] Key type .llcrypt registered [ 6246.598915] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 6263.987990] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 6265.399733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6265.454332] alg: No test for adler32 (adler32-zlib) [ 6266.584443] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 6266.847461] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 6268.521405] Key type lgssc registered [ 6269.662644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6305.673448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6305.722530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.967233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6307.001747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6307.109825] Lustre: lustre-MDT0000: new disk, initializing [ 6307.215157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.234391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6311.261646] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6321.768269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.932474] Lustre: 150283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6321.992393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6321.996780] Lustre: Skipped 1 previous similar message [ 6322.071720] Lustre: lustre-MDT0001: new disk, initializing [ 6322.214325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6322.249072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6322.265068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6326.783716] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6336.595470] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6342.403844] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6348.928793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.232865] Lustre: lustre-OST0000: new disk, initializing [ 6349.238872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6349.250580] Lustre: 152254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6349.354235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.331451] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6358.626621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6358.651247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6358.818424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6366.007251] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.306369] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_module obdecho/obdecho [ 6373.490214] LustreError: 153258:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6373.499302] LustreError: 153258:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6375.657360] LustreError: 153354:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6388.705933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6388.712376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.728279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6389.730873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6389.733622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.746508] Lustre: Skipped 1 previous similar message [ 6389.765114] Lustre: Skipped 1 previous similar message [ 6394.491127] Lustre: server umount lustre-MDT0000 complete [ 6398.770264] LustreError: 151081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799094 with bad export cookie 11095812201459226288 [ 6398.782159] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.802414] LustreError: 151081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6399.192577] Lustre: server umount lustre-MDT0001 complete [ 6412.879200] Lustre: server umount lustre-OST0000 complete [ 6439.396514] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 6451.582172] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 6488.646673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.796655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6488.837848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6488.900929] Lustre: lustre-MDT0000: new disk, initializing [ 6488.977118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.008269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6494.107540] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6504.641449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.760704] Lustre: 157077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6504.848054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6504.852572] Lustre: Skipped 1 previous similar message [ 6504.913274] Lustre: lustre-MDT0001: new disk, initializing [ 6504.988163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6505.015689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6505.033880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6508.981274] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6517.684886] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.681546] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6528.962076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6529.156422] Lustre: lustre-OST0000: new disk, initializing [ 6529.160227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6529.167475] Lustre: 159042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6529.231397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.707819] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6537.806225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6537.820127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6537.929461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6543.180280] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6547.383939] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6547.568304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6551.215612] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6551.409800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6553.571166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.573453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.581698] LustreError: Skipped 1 previous similar message [ 6553.593111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.593121] Lustre: Skipped 1 previous similar message [ 6553.623776] Lustre: Skipped 1 previous similar message [ 6558.695290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6558.702777] Lustre: Skipped 2 previous similar messages [ 6563.816587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6563.824761] Lustre: Skipped 1 previous similar message [ 6567.392476] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6567.551910] Lustre: server umount lustre-OST0000 complete [ 6574.053126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.057602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.068980] Lustre: Skipped 1 previous similar message [ 6574.081595] Lustre: Skipped 1 previous similar message [ 6578.002303] Lustre: server umount lustre-MDT0000 complete [ 6579.173550] LustreError: 157088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.198317] LustreError: 157088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6581.835090] LustreError: 157067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799277 with bad export cookie 11095812201459227247 [ 6581.835984] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6581.845574] LustreError: 157067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6582.199748] Lustre: server umount lustre-MDT0001 complete [ 6585.903687] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:54:39 (1787799279) [ 6587.753425] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6589.460260] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:54:42 (1787799282) [ 6597.177882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.564597] LustreError: 161457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.621105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.937717] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6602.726120] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.846039] LustreError: 161457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.632918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.074720] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6621.096575] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6626.357618] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.844548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.206468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.213236] Lustre: Skipped 1 previous similar message [ 6642.179390] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6651.612910] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6659.043337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.045259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6659.065894] Lustre: Skipped 2 previous similar messages [ 6659.084875] Lustre: Skipped 2 previous similar messages [ 6664.366567] Lustre: server umount lustre-MDT0000 complete [ 6667.479855] LustreError: 161452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.898448] LustreError: 161438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799364 with bad export cookie 11095812201459228129 [ 6668.916080] LustreError: 161438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6668.920093] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.282466] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.556623] Lustre: server umount lustre-MDT0001 complete [ 6684.796939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6698.480223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.777418] LustreError: 165298:0:(llog.c:1655:llog_backup()) MGC192.168.204.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6698.786814] Lustre: 165298:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6698.915868] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.936912] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6715.364387] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.381566] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6749.396943] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.421314] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6759.392376] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6759.400041] Lustre: 165298:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6800.362411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6805.008860] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6805.494979] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6813.695562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.692065] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.725584] LustreError: 165303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6819.320030] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6820.346775] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6827.639971] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6829.616884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6836.073285] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6837.505496] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6854.302446] Lustre: server umount lustre-MDT0000 complete [ 6856.161278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6856.164167] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.200808] Lustre: Skipped 1 previous similar message [ 6858.374149] LustreError: 164239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799553 with bad export cookie 11095812201459234583 [ 6858.387098] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6858.398631] LustreError: 164239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6858.911152] Lustre: server umount lustre-MDT0001 complete [ 6874.496194] Lustre: server umount lustre-OST0000 complete [ 6896.340317] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 6907.506683] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 6945.290517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.595574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6945.637803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6945.720869] Lustre: lustre-MDT0000: new disk, initializing [ 6945.788418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.794219] Lustre: Skipped 1 previous similar message [ 6945.811470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6949.619992] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6960.116761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.217299] Lustre: 170400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6960.493947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6964.663221] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6973.094197] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6978.328614] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6984.398230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.663450] Lustre: lustre-OST0000: new disk, initializing [ 6984.666251] Lustre: Skipped 1 previous similar message [ 6984.668881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6984.673520] Lustre: Skipped 2 previous similar messages [ 6984.678636] Lustre: 172379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6990.833148] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 6992.980846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6992.989172] Lustre: Skipped 1 previous similar message [ 6992.992583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6993.118813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6999.068073] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7003.165606] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7003.392502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7008.347890] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7008.572737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7013.864489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7013.866074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.876079] LustreError: Skipped 1 previous similar message [ 7013.882117] Lustre: Skipped 1 previous similar message [ 7013.885807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.885815] Lustre: Skipped 7 previous similar messages [ 7016.818422] Lustre: server umount lustre-OST0000 complete [ 7022.053880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7026.684876] Lustre: server umount lustre-MDT0000 complete [ 7029.225577] LustreError: 170412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.243459] LustreError: 170412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7030.114463] LustreError: 170392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799725 with bad export cookie 11095812201459251411 [ 7030.115198] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7030.125597] LustreError: 170392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7030.354841] Lustre: server umount lustre-MDT0001 complete [ 7040.749139] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:02:14 (1787799734) [ 7047.417806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.269833] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7061.790516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.183294] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7075.049585] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7080.787930] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7087.055762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.616173] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7101.091658] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7119.643174] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:03:32 (1787799812) [ 7127.526522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.528312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7127.554336] Lustre: Skipped 3 previous similar messages [ 7127.559545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7127.578084] Lustre: Skipped 5 previous similar messages [ 7130.289823] Lustre: server umount lustre-OST0000 complete [ 7140.178639] Lustre: server umount lustre-MDT0000 complete [ 7144.081427] LustreError: 174770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799839 with bad export cookie 11095812201459252293 [ 7144.081990] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.092908] LustreError: 174770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7144.475989] Lustre: server umount lustre-MDT0001 complete [ 7156.503854] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 7159.832222] Key type lgssc unregistered [ 7160.305510] LNet: 178540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7160.320953] LNetError: 178540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7160.353260] LNet: Removed LNI 192.168.204.138@tcp [ 7161.313977] Key type .llcrypt unregistered [ 7161.318057] Key type ._llcrypt unregistered [ 7181.306215] Key type ._llcrypt registered [ 7181.311706] Key type .llcrypt registered [ 7181.495818] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 7182.533297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7182.642110] alg: No test for adler32 (adler32-zlib) [ 7183.696736] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 7183.958713] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 7185.696962] Key type lgssc registered [ 7187.005918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7195.619631] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7195.640979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.296387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7216.417208] LustreError: 179784:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.138@tcp: failed processing log, type 4: rc = -110 [ 7221.536600] LustreError: 179784:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7221.547232] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7221.557797] LustreError: MGC192.168.204.138@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7221.572341] LustreError: 179784:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7221.592948] LustreError: 179784:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7221.604265] LustreError: 179784:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7221.730315] Lustre: server umount lustre-OST0000 complete [ 7221.733168] LustreError: 179784:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7231.040089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7232.627241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.835659] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7243.234647] LustreError: 180284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.252829] LustreError: 180284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7245.360381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.788355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7250.448382] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7257.912743] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7263.507828] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.179935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.473921] LustreError: 182099:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7270.490367] Lustre: 182099:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.138@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7270.722361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.592480] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7284.297790] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7286.753419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7286.769384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7286.794906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.291764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.304433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.329309] Lustre: Skipped 1 previous similar message [ 7292.754950] Lustre: server umount lustre-MDT0000 complete [ 7293.412084] LustreError: 180284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.430674] LustreError: 180284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7295.868207] LustreError: 180264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799991 with bad export cookie 356864814167809313 [ 7295.881767] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7296.109568] Lustre: server umount lustre-MDT0001 complete [ 7310.017411] Lustre: server umount lustre-OST0000 complete [ 7319.731268] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 7329.760417] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 7367.593132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.809451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7367.835705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7367.897865] Lustre: lustre-MDT0000: new disk, initializing [ 7367.966330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.986273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7372.468209] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7383.156600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.283363] Lustre: 185624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.347429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7383.350693] Lustre: Skipped 1 previous similar message [ 7383.431982] Lustre: lustre-MDT0001: new disk, initializing [ 7383.547936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.589698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7383.622414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7388.019113] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7396.684833] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7402.437929] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7408.345856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.550911] Lustre: lustre-OST0000: new disk, initializing [ 7408.554317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7408.561731] Lustre: 187591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7408.643434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.488099] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7415.359033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7415.381459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7415.556779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7424.825737] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7431.093101] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7431.402536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7436.965558] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7437.246856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7441.381343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.383723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.384844] LustreError: Skipped 1 previous similar message [ 7441.385660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7441.385664] Lustre: Skipped 2 previous similar messages [ 7441.431844] Lustre: Skipped 1 previous similar message [ 7446.080759] Lustre: server umount lustre-OST0000 complete [ 7451.618577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.624093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7451.637149] Lustre: Skipped 1 previous similar message [ 7451.650078] Lustre: Skipped 1 previous similar message [ 7456.745359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7456.754140] Lustre: Skipped 1 previous similar message [ 7457.569858] Lustre: server umount lustre-MDT0000 complete [ 7461.780906] LustreError: 185616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800157 with bad export cookie 356864814167810034 [ 7461.785150] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.787387] LustreError: 185616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7462.027823] Lustre: server umount lustre-MDT0001 complete [ 7475.967293] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:09:28 (1787800168) [ 7496.899605] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 7500.655526] Key type lgssc unregistered [ 7501.011124] LNet: 190543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7501.015047] LNetError: 190543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7501.029122] LNet: Removed LNI 192.168.204.138@tcp [ 7502.263534] Key type .llcrypt unregistered [ 7502.268190] Key type ._llcrypt unregistered [ 7523.937629] Key type ._llcrypt registered [ 7523.942737] Key type .llcrypt registered [ 7524.098443] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 7525.452233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7525.476955] alg: No test for adler32 (adler32-zlib) [ 7526.598370] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 7526.838959] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 7528.480314] Key type lgssc registered [ 7529.570397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7537.528153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7537.554941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.094271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.730692] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7549.411747] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.438354] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7552.620210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.092563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7557.693859] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7565.017591] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.403809] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7580.541588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.738711] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7580.743984] LustreError: 193626:0:(llog.c:1696:llog_backup()) MGC192.168.204.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7580.754935] Lustre: 193626:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.138@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7580.944596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.829913] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7600.052497] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.690600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.704291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.719757] Lustre: Skipped 2 previous similar messages [ 7603.731939] Lustre: Skipped 2 previous similar messages [ 7608.806680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7608.816615] Lustre: Skipped 2 previous similar messages [ 7608.958346] Lustre: server umount lustre-MDT0000 complete [ 7612.793128] LustreError: 191745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800308 with bad export cookie 6186226817379838418 [ 7612.808641] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.262425] Lustre: server umount lustre-MDT0001 complete [ 7628.365737] Lustre: server umount lustre-OST0000 complete [ 7639.505990] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 7650.651669] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 7685.660862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.855782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7685.874293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7685.930506] Lustre: lustre-MDT0000: new disk, initializing [ 7685.991207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.008130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7690.110302] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7700.181772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.350622] Lustre: 197149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7700.443348] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7700.447296] Lustre: Skipped 1 previous similar message [ 7700.457470] LustreError: 197932:0:(llog.c:1696:llog_backup()) MGC192.168.204.138@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7700.459401] LustreError: 197161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7700.466436] LustreError: 197932:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7700.495442] Lustre: 197932:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.138@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7700.519493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7700.527869] Lustre: Skipped 1 previous similar message [ 7700.645744] Lustre: lustre-MDT0001: new disk, initializing [ 7700.753432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7700.788971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7700.796993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7705.034224] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7713.984798] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7719.825738] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7725.767261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7725.992650] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7725.995276] Lustre: Skipped 1 previous similar message [ 7725.997759] LustreError: 199096:0:(llog.c:1696:llog_backup()) MGC192.168.204.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7726.003103] LustreError: 199096:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7726.006795] Lustre: 199096:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.138@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7726.012476] Lustre: 199096:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7726.026667] Lustre: lustre-OST0000: new disk, initializing [ 7726.030744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7726.035665] Lustre: 199120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7726.097605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.002895] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7732.289822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7732.303054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7732.386300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7740.772012] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7744.990986] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7745.238324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7749.418887] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7749.596711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7753.186628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7753.188201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.199939] LustreError: Skipped 1 previous similar message [ 7753.230831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7753.238278] Lustre: Skipped 1 previous similar message [ 7757.587498] Lustre: server umount lustre-OST0000 complete [ 7762.402854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7762.416795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.440740] Lustre: Skipped 1 previous similar message [ 7762.458856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7768.128488] Lustre: server umount lustre-MDT0000 complete [ 7768.566164] LustreError: 197157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.592638] LustreError: 197157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7772.717266] LustreError: 197162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.748085] LustreError: 197162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7773.912105] LustreError: 197944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800469 with bad export cookie 6186226817379839132 [ 7773.916780] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7773.932089] LustreError: 197944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7774.704095] Lustre: server umount lustre-MDT0001 complete [ 7784.596808] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:14:37 (1787800477) [ 7802.830415] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 7814.380696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7814.897891] LustreError: 202251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.967660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.345244] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7825.381413] LustreError: 202252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.408829] LustreError: 202252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7829.394849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.000412] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7836.917873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7843.058544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.456773] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7857.734959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.918953] Lustre: lustre-OST0001: new disk, initializing [ 7857.924404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7857.931600] Lustre: 204556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7858.030574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7858.036135] Lustre: Skipped 2 previous similar messages [ 7863.795835] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7867.955365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7867.969902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7868.074561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7873.083833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7878.917727] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7880.471238] Lustre: DEBUG MARKER: ADD OST3 [ 7884.461786] loop: module loaded [ 7884.566921] loop0: detected capacity change from 0 to 204800000 [ 7884.856423] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7884.866402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7884.885670] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7890.729240] loop0: detected capacity change from 0 to 204800000 [ 7891.937231] Lustre: DEBUG MARKER: STOP OST3 [ 7896.446223] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7896.620325] Lustre: lustre-OST0003: new disk, initializing [ 7896.624819] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7896.637247] Lustre: 206546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.328093] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7898.393274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7898.405759] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7898.459543] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7900.313504] Lustre: Failing over lustre-OST0003 [ 7900.384168] LustreError: 206732:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7900.466523] Lustre: server umount lustre-OST0003 complete [ 7902.747833] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7903.726904] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7903.733461] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7903.736495] LustreError: 205083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.743594] Lustre: Skipped 2 previous similar messages [ 7903.774842] LustreError: 205083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7910.862170] Lustre: DEBUG MARKER: START OST3 again [ 7917.489097] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.728938] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7919.210621] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7919.329911] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7919.331462] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7924.394288] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 7946.408861] Lustre: Failing over lustre-OST0003 [ 7946.634721] Lustre: server umount lustre-OST0003 complete [ 7948.786447] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.800662] Lustre: Skipped 1 previous similar message [ 7948.802622] LustreError: 203732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.832119] LustreError: 203732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7968.225560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7968.231957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.246831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7968.253790] Lustre: Skipped 2 previous similar messages [ 7971.994595] Lustre: server umount lustre-MDT0000 complete [ 7975.302506] LustreError: 205381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800670 with bad export cookie 6186226817379840014 [ 7975.310371] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.310913] LustreError: 205381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7979.501228] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7979.505016] Lustre: Skipped 4 previous similar messages [ 7981.874301] Lustre: server umount lustre-MDT0001 complete [ 7985.841664] Lustre: server umount lustre-OST0000 complete [ 7990.427987] Lustre: server umount lustre-OST0001 complete [ 7998.586761] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 8008.299482] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 8045.127734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.366194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8045.396140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8045.491577] Lustre: lustre-MDT0000: new disk, initializing [ 8045.627332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.634028] Lustre: Skipped 2 previous similar messages [ 8045.649814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8050.094837] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8061.287848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.464324] Lustre: 212013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8061.542330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8061.547923] Lustre: Skipped 1 previous similar message [ 8061.661311] Lustre: lustre-MDT0001: new disk, initializing [ 8061.772093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8061.782856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8066.206542] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8075.915873] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8080.803445] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8086.437928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8086.686840] Lustre: 213982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8088.445306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8088.588460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8092.668953] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8100.831957] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8105.520824] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8105.704117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8109.961297] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8110.172765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8114.164886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.172754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.173912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.182313] Lustre: Skipped 6 previous similar messages [ 8114.204761] Lustre: Skipped 3 previous similar messages [ 8118.180942] Lustre: server umount lustre-OST0000 complete [ 8123.362470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8123.370176] LustreError: Skipped 1 previous similar message [ 8127.798400] Lustre: server umount lustre-MDT0000 complete [ 8129.506052] LustreError: 212887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.529190] LustreError: 212887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8131.238534] LustreError: 213981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800826 with bad export cookie 6186226817379842317 [ 8131.239971] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.262153] LustreError: 213981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8131.507811] Lustre: server umount lustre-MDT0001 complete [ 8134.595496] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:20:28 (1787800828) [ 8149.615225] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 8159.342806] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 8198.863751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.076780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8199.129356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8199.141615] Lustre: Skipped 1 previous similar message [ 8199.285978] Lustre: lustre-MDT0000: new disk, initializing [ 8199.295609] Lustre: Skipped 1 previous similar message [ 8199.470093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8199.479512] Lustre: Skipped 2 previous similar messages [ 8199.529514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8199.541513] Lustre: Skipped 1 previous similar message [ 8204.353539] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8215.699801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.881592] Lustre: 218590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8216.473892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8222.182878] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8234.871631] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8243.024574] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8250.883739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.130206] Lustre: 220561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8253.110587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8253.305309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8258.707409] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8269.414588] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8274.148189] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8274.385817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8279.261390] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8279.507112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8284.130054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.130942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.142598] LustreError: Skipped 1 previous similar message [ 8284.143543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.143548] Lustre: Skipped 3 previous similar messages [ 8284.188553] Lustre: Skipped 3 previous similar messages [ 8287.492224] Lustre: server umount lustre-OST0000 complete [ 8298.007727] Lustre: server umount lustre-MDT0000 complete [ 8299.490921] LustreError: 218599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.510925] LustreError: 218599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8302.440642] LustreError: 218582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800997 with bad export cookie 6186226817379843199 [ 8302.457513] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.467923] LustreError: 218582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8303.043494] Lustre: server umount lustre-MDT0001 complete [ 8314.159312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8319.041866] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8327.359298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.674607] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8338.832985] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8345.065896] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8351.170683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.636844] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8364.383874] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8375.287724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8375.295905] LustreError: Skipped 1 previous similar message [ 8375.299649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.319512] Lustre: Skipped 2 previous similar messages [ 8375.340416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8375.354223] Lustre: Skipped 5 previous similar messages [ 8379.981723] Lustre: server umount lustre-OST0000 complete [ 8390.539751] Lustre: server umount lustre-MDT0000 complete [ 8394.487861] LustreError: 222761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801090 with bad export cookie 6186226817379844081 [ 8394.504545] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.986454] Lustre: server umount lustre-MDT0001 complete [ 8501.651327] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 8504.388028] Key type lgssc unregistered [ 8504.676801] LNet: 226446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8504.684937] LNetError: 226446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8504.718247] LNet: Removed LNI 192.168.204.138@tcp [ 8505.596154] Key type .llcrypt unregistered [ 8505.600726] Key type ._llcrypt unregistered [ 8528.512119] Key type ._llcrypt registered [ 8528.514293] Key type .llcrypt registered [ 8528.641807] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 8547.852480] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 8550.037268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8550.093139] alg: No test for adler32 (adler32-zlib) [ 8551.286389] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 8551.492546] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 8553.128175] Key type lgssc registered [ 8554.613744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8589.704680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8589.726211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.988153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8591.025866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8591.087501] Lustre: lustre-MDT0000: new disk, initializing [ 8591.150315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.174479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8595.216593] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8605.333914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.484385] Lustre: 229799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8605.573089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8605.577944] Lustre: Skipped 1 previous similar message [ 8605.667289] Lustre: lustre-MDT0001: new disk, initializing [ 8605.783742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.824819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8605.836796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8611.038723] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8619.480330] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8625.200521] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8632.544908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8632.804083] Lustre: lustre-OST0000: new disk, initializing [ 8632.811810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8632.820668] Lustre: 231769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8632.925759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8638.533562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8638.546139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8638.653596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8639.101152] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8648.093060] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.486475] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8652.740573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8657.449516] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8657.682914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8664.545790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8664.555135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.561598] LustreError: Skipped 1 previous similar message [ 8664.567090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8664.577602] Lustre: Skipped 1 previous similar message [ 8666.507836] Lustre: server umount lustre-OST0000 complete [ 8669.665972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.675026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.677399] Lustre: Skipped 1 previous similar message [ 8674.786220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8675.768758] Lustre: server umount lustre-MDT0000 complete [ 8679.255364] LustreError: 229792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801374 with bad export cookie 7016000046068725693 [ 8679.263031] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8679.267310] LustreError: 229792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8679.706939] Lustre: server umount lustre-MDT0001 complete [ 8701.449145] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 8715.323042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.940390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.091771] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8721.381205] LustreError: 234705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8721.411254] LustreError: 234705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8726.504092] LustreError: 234704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.986427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.445472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.618809] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8741.445280] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8749.637547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8750.263517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.964782] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8770.169214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8770.395836] Lustre: lustre-OST0001: new disk, initializing [ 8770.406319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8770.413571] Lustre: 237017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8770.534354] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8777.369167] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8778.880796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8778.887966] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8779.044379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8789.587287] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8802.194221] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8815.795609] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:31:48 (1787801508) [ 8818.659777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8818.669285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.692227] Lustre: Skipped 1 previous similar message [ 8818.696398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8818.704559] Lustre: Skipped 1 previous similar message [ 8824.086402] Lustre: server umount lustre-MDT0000 complete [ 8825.317387] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8825.335907] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8827.743541] LustreError: 237016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801523 with bad export cookie 7016000046068726575 [ 8827.754281] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.758570] LustreError: 237016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8828.094752] Lustre: server umount lustre-MDT0001 complete [ 8842.489352] Lustre: server umount lustre-OST0000 complete [ 8845.793130] Lustre: 228026:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787801525/real 1787801525] req@ffff8c3b17edc780 x1874645480489728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787801541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8845.838453] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.861888] Lustre: Skipped 3 previous similar messages [ 8846.410762] Lustre: server umount lustre-OST0001 complete [ 8853.847741] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 8866.600910] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 8904.374317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.632181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8904.678527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8904.779324] Lustre: lustre-MDT0000: new disk, initializing [ 8904.865683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.882771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8909.578577] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8919.132741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.262805] Lustre: 240963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8919.311352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8919.315495] Lustre: Skipped 1 previous similar message [ 8919.375756] Lustre: lustre-MDT0001: new disk, initializing [ 8919.517104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8919.535983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8923.278776] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8931.800626] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8937.149563] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8943.166537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8943.408672] Lustre: lustre-OST0000: new disk, initializing [ 8943.416391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8943.428300] Lustre: 242931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8945.348048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8945.358374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8945.455838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8949.040127] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 8958.018977] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.477535] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8963.661395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8967.985143] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8968.290391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8971.233472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8971.236605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8971.253220] LustreError: Skipped 1 previous similar message [ 8971.255989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8971.256026] Lustre: Skipped 4 previous similar messages [ 8971.290554] Lustre: Skipped 1 previous similar message [ 8976.388571] Lustre: server umount lustre-OST0000 complete [ 8980.970858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8980.987412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.000602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8981.006346] Lustre: Skipped 1 previous similar message [ 8986.814343] Lustre: server umount lustre-MDT0000 complete [ 8990.690738] LustreError: 241852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8990.708053] LustreError: 241852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8990.944133] LustreError: 240955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801686 with bad export cookie 7016000046068728710 [ 8990.948972] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8990.958625] LustreError: 240955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8991.426190] Lustre: server umount lustre-MDT0001 complete [ 9000.635618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9001.083842] LustreError: 245147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9001.193838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.202790] Lustre: Skipped 2 previous similar messages [ 9005.798159] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9010.662710] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9010.689721] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9015.276433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.944609] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9027.104900] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9032.633672] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9039.507287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.297774] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9052.873379] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9199.588959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.591626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9199.623286] Lustre: Skipped 1 previous similar message [ 9199.631282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.655531] Lustre: Skipped 5 previous similar messages [ 9203.980910] Lustre: server umount lustre-OST0000 complete [ 9222.112144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9222.316768] Lustre: server umount lustre-MDT0000 complete [ 9225.188205] LustreError: 245143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.211917] LustreError: 245143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9225.949771] LustreError: 246993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801921 with bad export cookie 7016000046068729585 [ 9225.949771] LustreError: 245128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801921 with bad export cookie 7016000046068729585 [ 9225.950111] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9226.379540] Lustre: server umount lustre-MDT0001 complete [ 9322.533459] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [ 9326.490957] Key type lgssc unregistered [ 9326.973702] LNet: 248761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9326.987600] LNetError: 248761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9327.006208] LNet: Removed LNI 192.168.204.138@tcp [ 9328.279174] Key type .llcrypt unregistered [ 9328.289249] Key type ._llcrypt unregistered [ 9356.184276] Key type ._llcrypt registered [ 9356.189469] Key type .llcrypt registered [ 9356.297427] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 9375.477533] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 9377.306356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9377.343687] alg: No test for adler32 (adler32-zlib) [ 9378.511927] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [ 9378.958651] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 9380.744419] Key type lgssc registered [ 9382.231847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9437.571288] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 9448.250504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9448.280376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.646690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.706352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9449.757962] Lustre: lustre-MDT0000: new disk, initializing [ 9449.838651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.852495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9455.447275] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9468.462084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.643244] Lustre: 252835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9468.727243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9468.731116] Lustre: Skipped 1 previous similar message [ 9468.815145] Lustre: lustre-MDT0001: new disk, initializing [ 9468.913891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9468.949906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9468.959068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9474.133721] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9479.447911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9486.762236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.185203] Lustre: lustre-OST0000: new disk, initializing [ 9487.188534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9487.193840] Lustre: 254469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9487.287327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.779904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9488.813199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9488.980322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9493.742746] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9506.544362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9506.916739] Lustre: lustre-OST0001: new disk, initializing [ 9506.931694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9506.957443] Lustre: 255339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9507.163986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9516.251446] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9516.252157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9516.276888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9516.585642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9530.753092] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9536.657582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9550.817363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9550.822942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.836451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.361752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.363623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9555.432800] Lustre: server umount lustre-MDT0000 complete [ 9559.036950] LustreError: 253639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802254 with bad export cookie 7748545458654513041 [ 9559.041653] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9559.047387] LustreError: 253639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9559.508612] Lustre: server umount lustre-MDT0001 complete [ 9572.692231] Lustre: server umount lustre-OST0000 complete [ 9585.723902] Lustre: server umount lustre-OST0001 complete [ 9599.029478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9608.887165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9617.728163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9627.619346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9647.771649] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 9659.164884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.179658] Lustre: lustre-MDT0000: reset Object Index mappings [ 9659.607169] LustreError: 260351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.624426] LustreError: 260351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9659.706231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.701677] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9665.011433] LustreError: 260352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.069818] LustreError: 260351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.115940] LustreError: 260352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.791608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9673.809448] Lustre: lustre-MDT0001: reset Object Index mappings [ 9674.211164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9678.988309] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9682.072502] Lustre: 261493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9688.938719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9694.961347] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9699.507517] LustreError: 261848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.534741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9704.760842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9704.770455] Lustre: Skipped 1 previous similar message [ 9705.799785] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9705.813843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9710.911858] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9717.766152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9721.414401] Lustre: 263368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9726.445082] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.455844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9726.460224] Lustre: Skipped 4 previous similar messages [ 9726.468787] Lustre: Skipped 4 previous similar messages [ 9731.559678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.570124] Lustre: Skipped 5 previous similar messages [ 9731.907679] Lustre: server umount lustre-MDT0000 complete [ 9736.095538] LustreError: 260332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802431 with bad export cookie 7748545458654519824 [ 9736.105654] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9736.109703] LustreError: 260332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9736.422800] Lustre: server umount lustre-MDT0001 complete [ 9751.132601] Lustre: server umount lustre-OST0000 complete [ 9753.057075] Lustre: 250342:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787802432/real 1787802432] req@ffff8c3b289afc00 x1874646347676544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787802448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9753.100504] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.124050] Lustre: Skipped 1 previous similar message [ 9756.791800] Lustre: server umount lustre-OST0001 complete [ 9776.743639] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:47:50 (1787802470) [ 9792.631455] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 9802.240649] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 9835.647225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.872697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9835.910204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9835.979320] Lustre: lustre-MDT0000: new disk, initializing [ 9836.061591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.080802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9840.077816] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9849.504526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.665255] Lustre: 267621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9849.813906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9849.822363] Lustre: Skipped 1 previous similar message [ 9849.960262] Lustre: lustre-MDT0001: new disk, initializing [ 9850.125667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9850.138665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9854.907493] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9863.854487] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9869.264975] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9875.864914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.197176] Lustre: lustre-OST0000: new disk, initializing [ 9876.201200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9876.210074] Lustre: 269586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9877.805303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9877.811899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9877.912695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9881.413573] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9890.324753] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9894.257504] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9894.441797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9898.143858] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9898.339836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9903.590256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.608594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9903.609773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9903.625459] Lustre: Skipped 1 previous similar message [ 9906.360088] Lustre: server umount lustre-OST0000 complete [ 9911.777227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9911.789969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9911.803066] Lustre: Skipped 1 previous similar message [ 9911.809947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9915.966128] Lustre: server umount lustre-MDT0000 complete [ 9918.946241] LustreError: 267631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9918.960056] LustreError: 267631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9919.453255] LustreError: 267612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802614 with bad export cookie 7748545458654521924 [ 9919.459651] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9919.464509] LustreError: 267612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9919.896618] Lustre: server umount lustre-MDT0001 complete [ 9940.159090] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 9951.598791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.194583] LustreError: 272520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.266891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.282644] Lustre: Skipped 2 previous similar messages [ 9957.035777] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9965.552305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.418405] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9973.298843] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9980.619535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9987.534873] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [ 9996.383533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9996.608790] Lustre: lustre-OST0001: new disk, initializing [ 9996.619815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9996.631865] Lustre: 274832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9998.116703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9998.130110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9998.285605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10002.737913] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10011.655506] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10015.080840] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10019.917173] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10019.997149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10027.492224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10027.503465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.510801] Lustre: Skipped 1 previous similar message [10027.514668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10027.518846] Lustre: Skipped 2 previous similar messages [10040.293090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10040.483803] Lustre: server umount lustre-MDT0000 complete [10044.133461] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802739 with bad export cookie 7748545458654522806 [10044.139821] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.145646] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10044.391406] LustreError: 273545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.408066] LustreError: 273545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10044.566714] Lustre: server umount lustre-MDT0001 complete [10050.670483] Lustre: server umount lustre-OST0000 complete [10054.315309] Lustre: server umount lustre-OST0001 complete [10069.618345] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [10079.734417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.593275] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10092.779412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.287262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10093.293521] Lustre: Skipped 4 previous similar messages [10098.302468] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10101.367295] Lustre: 279024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10108.432930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10114.983752] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10118.017097] LustreError: 279378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.052124] LustreError: 279378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10123.519185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.862320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10124.864296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10128.798318] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10135.438306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10138.331320] Lustre: 280901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10141.982656] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10141.993991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10150.914925] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:54:03 (1787802843) [10170.339115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.343771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.362895] Lustre: Skipped 5 previous similar messages [10170.375106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.384506] Lustre: Skipped 12 previous similar messages [10172.631841] Lustre: server umount lustre-MDT0000 complete [10175.833527] LustreError: 277866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802871 with bad export cookie 7748545458654529148 [10175.835952] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10175.845192] LustreError: 277866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10190.306132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10190.618860] Lustre: server umount lustre-MDT0001 complete [10200.646217] Lustre: server umount lustre-OST0000 complete [10211.011674] Lustre: server umount lustre-OST0001 complete [10218.726240] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [10229.336626] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [10274.867430] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [10283.859178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.218256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10284.269511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10284.357422] Lustre: lustre-MDT0000: new disk, initializing [10284.490349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10289.161469] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10300.414308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.560815] Lustre: 285149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10300.920304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10306.470706] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10321.207028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.525915] Lustre: 286784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10322.859941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10328.030808] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10339.857942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10340.130350] Lustre: 287655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10341.941203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10341.947907] Lustre: Skipped 1 previous similar message [10342.060600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10349.491544] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10362.481878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10367.465441] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10367.474984] Lustre: Skipped 1 previous similar message [10384.330285] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:57:57 (1787803077) [10387.937890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10387.946838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.956952] Lustre: Skipped 5 previous similar messages [10387.963191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.971485] Lustre: Skipped 9 previous similar messages [10388.198379] Lustre: server umount lustre-MDT0000 complete [10388.462474] LustreError: 285157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.473188] LustreError: 285157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10391.818434] LustreError: 285946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803087 with bad export cookie 7748545458654537023 [10391.827543] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10391.834361] LustreError: 285946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10392.050380] Lustre: server umount lustre-MDT0001 complete [10407.059577] Lustre: server umount lustre-OST0000 complete [10409.954917] Lustre: 250342:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787803089/real 1787803089] req@ffff8c3a03da5e00 x1874646347927680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787803105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10410.328263] Lustre: server umount lustre-OST0001 complete [10419.606257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.365316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10420.376993] Lustre: Skipped 6 previous similar messages [10425.028672] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10435.175216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.907459] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10447.717358] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10453.166452] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10459.717106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10465.806585] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10473.256867] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10492.992117] loop1: detected capacity change from 0 to 204800000 [10493.228866] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10493.241878] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10493.258637] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10498.992529] loop1: detected capacity change from 0 to 204800000 [10504.513124] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.607061] LustreError: 289891:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs11433: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10504.630981] LustreError: 289891:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs11433 [10504.637223] LustreError: 289891:0:(mgs_handler.c:517:mgs_target_reg()) fs11433-MDT0001: Error checking target index: rc = -13 [10504.652067] LustreError: fs11433-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10504.662600] LustreError: 293621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10504.669762] LustreError: 293621:0:(tgt_mount.c:2129:server_put_super()) no obd fs11433-MDT0001 [10504.674101] LustreError: 293621:0:(tgt_mount.c:116:server_deregister_mount()) fs11433-MDT0001 not registered [10504.708302] Lustre: server umount fs11433-MDT0001 complete [10504.710515] LustreError: 293621:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10513.116962] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.244302] Lustre: Setting parameter fs11433-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11433-MDT0001 [10513.305692] Lustre: srv-fs11433-MDT0001: No data found on store. Initialize space. [10513.314570] Lustre: Skipped 4 previous similar messages [10513.397299] Lustre: fs11433-MDT0001: new disk, initializing [10513.402798] Lustre: Skipped 3 previous similar messages [10513.532715] Lustre: srv-fs11433-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10513.540334] Lustre: 294089:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11433-MDT0001-mdtlov: can't create new llog: rc = -115 [10513.549914] LustreError: 294089:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11433-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10518.225899] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10525.597493] Lustre: server umount fs11433-MDT0001 complete [10539.960707] loop1: detected capacity change from 0 to 204800000 [10546.031782] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.542976] Lustre: 295657:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11433-MDT0001-mdtlov: can't create new llog: rc = -115 [10546.558061] LustreError: 295657:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11433-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10547.178243] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10551.442627] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10556.541231] Lustre: server umount fs11433-MDT0001 complete [10567.650267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.666719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.674198] Lustre: Skipped 6 previous similar messages [10567.699203] Lustre: Skipped 2 previous similar messages [10572.856208] Lustre: server umount lustre-MDT0000 complete [10581.847495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.031367] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.928951] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10587.634586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10595.354976] loop1: detected capacity change from 0 to 204800000 [10601.239873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.806428] Lustre: 298117:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11433-MDT0001-mdtlov: can't create new llog: rc = -115 [10601.822155] LustreError: 298117:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11433-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10606.250824] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10611.238396] Lustre: server umount fs11433-MDT0001 complete [10638.824623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10638.850456] LustreError: Skipped 1 previous similar message [10642.688978] Lustre: server umount lustre-OST0000 complete [10654.181506] LustreError: 289902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10654.212732] LustreError: 289902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 113 previous similar messages [10654.379932] Lustre: server umount lustre-MDT0000 complete [10658.488251] LustreError: 289884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803354 with bad export cookie 7748545458654539711 [10658.490497] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10658.501660] LustreError: 289884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10658.878631] Lustre: server umount lustre-MDT0001 complete [10672.383481] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [10675.956844] Key type lgssc unregistered [10676.348689] LNet: 300467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10676.362511] LNetError: 300467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10676.377906] LNet: Removed LNI 192.168.204.138@tcp [10677.421202] Key type .llcrypt unregistered [10677.428877] Key type ._llcrypt unregistered [10697.407462] Key type ._llcrypt registered [10697.413522] Key type .llcrypt registered [10697.491299] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [10698.959476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10698.989666] alg: No test for adler32 (adler32-zlib) [10700.072856] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [10700.320913] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [10701.976177] Key type lgssc registered [10703.219696] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.856447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10710.871634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.368748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.055509] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10721.771182] LustreError: 301739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10721.791268] LustreError: 301739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10726.882301] LustreError: 301738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10727.123587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.517075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10731.282660] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10737.376643] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10743.637848] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10752.080088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10752.594148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10756.731980] LustreError: 303567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10761.777669] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10762.227452] LustreError: 303753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.337506] LustreError: 303567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.735976] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10777.587988] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.616784] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10795.260100] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.289881] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10795.574705] loop1: detected capacity change from 0 to 204800000 [10795.909941] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10795.947956] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10795.982683] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10803.026375] loop1: detected capacity change from 0 to 204800000 [10809.127457] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.285508] Lustre: fs17928-OST0000: new disk, initializing [10809.293267] Lustre: srv-fs17928-OST0000: No data found on store. Initialize space. [10809.307465] Lustre: 305473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10809.435928] Lustre: fs17928-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10816.258910] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10825.041452] Lustre: server umount fs17928-OST0000 complete [10828.771953] LustreError: 303568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.827848] LustreError: 303568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10839.044434] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10843.345610] loop1: detected capacity change from 0 to 204800000 [10850.870396] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.236579] Lustre: fs17928-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10854.379671] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10858.902908] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10869.551645] loop2: detected capacity change from 0 to 204800000 [10869.872386] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.895761] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.917765] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10875.812361] loop2: detected capacity change from 0 to 204800000 [10881.843841] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10881.989812] LustreError: fs17928-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10882.001566] LustreError: fs17928-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10882.011385] LustreError: 308514:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10882.017730] LustreError: 308514:0:(tgt_mount.c:2129:server_put_super()) no obd fs17928-OST0001 [10882.022863] LustreError: 308514:0:(tgt_mount.c:116:server_deregister_mount()) fs17928-OST0001 not registered [10882.045666] Lustre: server umount fs17928-OST0001 complete [10882.048337] LustreError: 308514:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10894.444291] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.587652] Lustre: fs17928-OST0001: new disk, initializing [10894.598264] Lustre: srv-fs17928-OST0001: No data found on store. Initialize space. [10894.612468] Lustre: 309064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10894.711354] Lustre: fs17928-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10895.333923] LustreError: 303753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.355397] LustreError: 303753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10903.126860] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10914.782704] Lustre: server umount fs17928-OST0000 complete [10926.694852] Lustre: server umount fs17928-OST0001 complete [10941.206628] loop1: detected capacity change from 0 to 204800000 [10949.823208] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.277774] Lustre: fs17928-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10958.726521] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10966.632757] loop2: detected capacity change from 0 to 204800000 [10973.123280] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10981.586750] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [10992.409405] Lustre: server umount fs17928-OST0000 complete [10992.609802] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11001.956507] Lustre: server umount fs17928-OST0001 complete [11008.004403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11018.208962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11018.213391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11018.241148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11018.282714] Lustre: Skipped 1 previous similar message [11018.306707] Lustre: Skipped 1 previous similar message [11022.549990] Lustre: server umount lustre-OST0000 complete [11028.449427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11028.454819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11028.472481] Lustre: Skipped 1 previous similar message [11028.493241] Lustre: Skipped 1 previous similar message [11033.577866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11033.598576] Lustre: Skipped 1 previous similar message [11037.669816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11037.679957] Lustre: Skipped 1 previous similar message [11042.272151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11042.538728] Lustre: server umount lustre-MDT0000 complete [11042.786908] LustreError: 301734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.807544] LustreError: 301734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 73 previous similar messages [11046.578692] LustreError: 301719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803742 with bad export cookie 2720576050381984615 [11046.590378] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11046.600908] LustreError: 301719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11047.176603] Lustre: server umount lustre-MDT0001 complete [11058.392426] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [11061.411670] Key type lgssc unregistered [11061.730652] LNet: 315000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11061.750141] LNetError: 315000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11061.786470] LNet: Removed LNI 192.168.204.138@tcp [11062.723283] Key type .llcrypt unregistered [11062.725565] Key type ._llcrypt unregistered [11098.176780] Key type ._llcrypt registered [11098.188203] Key type .llcrypt registered [11098.360647] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [11117.297470] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [11118.697238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11118.866356] alg: No test for adler32 (adler32-zlib) [11119.958709] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [11120.218567] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [11121.945152] Key type lgssc registered [11123.356916] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.375346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.401581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.660694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11170.698569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11170.819332] Lustre: lustre-MDT0000: new disk, initializing [11170.951201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11170.978353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11176.307881] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11187.050775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11187.200688] Lustre: 318862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11187.266884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11187.271299] Lustre: Skipped 1 previous similar message [11187.341350] Lustre: lustre-MDT0001: new disk, initializing [11187.449274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11187.491290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11187.510131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11191.823823] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11200.173520] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11205.788939] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11215.636609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.416700] Lustre: lustre-OST0000: new disk, initializing [11216.428650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11216.445938] Lustre: 320834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11216.681663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11225.725033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11225.746985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11225.898736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11227.000174] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11240.134482] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11243.576994] Lustre: Mounted lustre-client - version 2.17.57_57_g44c575c [11244.983504] Lustre: Unmounted lustre-client [11261.921360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11261.928179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.941297] LustreError: Skipped 1 previous similar message [11261.952428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.966280] Lustre: Skipped 1 previous similar message [11261.990328] Lustre: Skipped 1 previous similar message [11263.980700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11263.999020] Lustre: Skipped 1 previous similar message [11266.100851] Lustre: server umount lustre-OST0000 complete [11274.212563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.223985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11274.251793] Lustre: Skipped 1 previous similar message [11274.278282] Lustre: Skipped 1 previous similar message [11278.929857] Lustre: server umount lustre-MDT0000 complete [11284.451214] LustreError: 318870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.488908] LustreError: 318870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11284.889455] LustreError: 318854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803980 with bad export cookie 4382572371624236787 [11284.906805] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.919726] LustreError: 318854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11285.510139] Lustre: server umount lustre-MDT0001 complete [11296.830465] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [11299.733404] Key type lgssc unregistered [11303.201610] LNet: 323000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11303.218497] LNetError: 323000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11375.913939] LNet: Removed LNI 192.168.204.138@tcp [11376.790664] Key type .llcrypt unregistered [11376.793749] Key type ._llcrypt unregistered [11396.524721] Key type ._llcrypt registered [11396.531166] Key type .llcrypt registered [11396.746941] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [11413.121261] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [11414.841585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.041511] alg: No test for adler32 (adler32-zlib) [11416.164353] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [11416.545459] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [11418.272242] Key type lgssc registered [11419.574977] Lustre: Echo OBD driver; http://www.lustre.org/ [11452.398053] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:15:45 (1787804145) [11461.015575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11461.029546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.338348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11462.384658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11462.491718] Lustre: lustre-MDT0000: new disk, initializing [11462.617819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11462.715945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11467.748443] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11478.476617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.630422] Lustre: 326224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11478.731261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11478.736145] Lustre: Skipped 1 previous similar message [11478.901432] Lustre: lustre-MDT0001: new disk, initializing [11479.045943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11479.101127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11479.113756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11484.567727] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11495.921760] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11503.107307] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11513.487076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11513.722324] Lustre: lustre-OST0000: new disk, initializing [11513.725118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11513.730669] Lustre: 328196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11513.877498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11521.159878] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11523.225882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11523.242222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11523.402929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11530.643890] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11544.037324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11544.037724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.046230] LustreError: Skipped 1 previous similar message [11544.063559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11544.070645] Lustre: Skipped 1 previous similar message [11549.156223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.170732] Lustre: Skipped 1 previous similar message [11549.514919] Lustre: server umount lustre-OST0000 complete [11554.274433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.284674] Lustre: Skipped 1 previous similar message [11554.289249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11554.300385] Lustre: Skipped 1 previous similar message [11558.937317] Lustre: server umount lustre-MDT0000 complete [11562.387207] LustreError: 326217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804257 with bad export cookie 10696463567130886056 [11562.393473] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.400066] LustreError: 326217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11562.813356] Lustre: server umount lustre-MDT0001 complete [11573.908901] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [11576.787715] Key type lgssc unregistered [11577.070608] LNet: 330108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11577.079071] LNetError: 330108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11577.091485] LNet: Removed LNI 192.168.204.138@tcp [11577.787987] Key type .llcrypt unregistered [11577.791159] Key type ._llcrypt unregistered [11598.799470] Key type ._llcrypt registered [11598.803297] Key type .llcrypt registered [11598.964032] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [11600.190459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11600.234556] alg: No test for adler32 (adler32-zlib) [11601.420353] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [11601.675138] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [11603.344772] Key type lgssc registered [11604.402206] Lustre: Echo OBD driver; http://www.lustre.org/ [11612.256733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11612.278585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.753703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11618.017710] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11623.915258] LustreError: 331381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11623.944430] LustreError: 331381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11627.006755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.413989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11631.521813] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11640.602331] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11646.989166] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11653.615980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11653.965890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11659.957366] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11667.290167] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11680.475752] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11682.083921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11687.908290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.925718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.224058] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [11690.229473] Lustre: Skipped 2 previous similar messages [11691.461750] Lustre: server umount lustre-MDT0000 complete [11693.028394] LustreError: 331381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.047941] LustreError: 331381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11694.355617] LustreError: 331363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804389 with bad export cookie 3969510429743242927 [11694.360565] LustreError: 331363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11694.361482] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11695.328691] LustreError: 331377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.346847] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.38@tcp (stopping) [11698.146320] LustreError: 331378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.164274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.183829] Lustre: Skipped 2 previous similar messages [11698.190059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11700.978264] Lustre: server umount lustre-MDT0001 complete [11705.235892] Lustre: server umount lustre-OST0000 complete [11713.410191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11717.215544] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11719.810806] Lustre: 335559:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.138@tcp [11721.611944] Lustre: server umount MGS complete [11729.511822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.783437] LustreError: 336131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.796881] LustreError: 336131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11729.842692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11734.520556] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11740.130780] LustreError: 336132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.144472] LustreError: 336132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11742.094134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.630186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11746.171393] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11752.903751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.230488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.031892] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11763.784541] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11765.002784] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11769.584603] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11770.718658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11775.969065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11775.974735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11775.991870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11775.994731] Lustre: Skipped 2 previous similar messages [11778.024188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11778.026757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11788.256559] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11788.268237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11788.278455] Lustre: Skipped 5 previous similar messages [11788.387467] Lustre: server umount lustre-OST0000 complete [11793.382973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.399100] Lustre: Skipped 1 previous similar message [11797.469254] Lustre: server umount lustre-MDT0000 complete [11798.497743] LustreError: 336128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11798.517685] LustreError: 336128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11800.339533] LustreError: 336114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804495 with bad export cookie 3969510429743243767 [11800.346157] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11800.617779] Lustre: server umount lustre-MDT0001 complete [11810.879308] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [11814.271249] Key type lgssc unregistered [11814.563968] LNet: 339571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11814.574468] LNetError: 339571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11814.594799] LNet: Removed LNI 192.168.204.138@tcp [11815.589223] Key type .llcrypt unregistered [11815.591137] Key type ._llcrypt unregistered [11832.673774] Key type ._llcrypt registered [11832.676381] Key type .llcrypt registered [11832.774135] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [11843.873572] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [11844.501578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11844.524190] alg: No test for adler32 (adler32-zlib) [11845.511126] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [11845.668355] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [11847.304173] Key type lgssc registered [11848.063799] Lustre: Echo OBD driver; http://www.lustre.org/ [11871.531587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11871.544460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11872.689419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11872.716430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11872.781230] Lustre: lustre-MDT0000: new disk, initializing [11872.830828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11872.844127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11876.283848] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11884.262281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.351084] Lustre: 342725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11884.403203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11884.407300] Lustre: Skipped 1 previous similar message [11884.464220] Lustre: lustre-MDT0001: new disk, initializing [11884.521165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11884.543055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11884.551497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11888.730192] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11895.969801] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11899.919786] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11905.100411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11905.341749] Lustre: lustre-OST0000: new disk, initializing [11905.347066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11905.354058] Lustre: 344692:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11905.419967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.457394] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [11914.286644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11914.296414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11914.403225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11916.064805] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11919.908186] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11920.044787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11923.911501] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11924.123709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11929.569148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11929.579622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.593974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11930.087482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11930.095210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.123266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11930.125827] Lustre: Skipped 1 previous similar message [11935.204499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.215148] Lustre: Skipped 1 previous similar message [11940.320116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11940.329823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11940.336571] Lustre: Skipped 1 previous similar message [11940.467397] Lustre: server umount lustre-OST0000 complete [11945.443342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.446481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.454250] Lustre: Skipped 1 previous similar message [11955.682164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11955.684394] Lustre: Skipped 4 previous similar messages [11957.216234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11957.368544] Lustre: server umount lustre-MDT0000 complete [11959.961246] LustreError: 342716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804655 with bad export cookie 7804708812380965691 [11959.961938] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11959.974198] LustreError: 342716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11960.171216] Lustre: server umount lustre-MDT0001 complete [11966.257665] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:24:20 (1787804660) [11979.919397] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [11988.372444] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [12015.313943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12015.536807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12015.556058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12015.613769] Lustre: lustre-MDT0000: new disk, initializing [12015.669677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12015.682503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12018.901410] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12027.550871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.636808] Lustre: 349451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12027.687620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12027.691179] Lustre: Skipped 1 previous similar message [12027.764630] Lustre: lustre-MDT0001: new disk, initializing [12027.828582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12027.860904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12027.869910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12031.462059] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12039.218974] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12043.654685] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12049.907267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12050.068328] Lustre: lustre-OST0000: new disk, initializing [12050.071695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12050.076706] Lustre: 351419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12050.146443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12056.663955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12056.676035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12056.789012] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12057.009531] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12068.376338] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12103.144934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12103.145650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12103.149799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12103.159089] Lustre: Skipped 1 previous similar message [12103.181516] LustreError: Skipped 1 previous similar message [12106.075454] Lustre: server umount lustre-OST0000 complete [12109.793336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12109.799231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.105749] Lustre: server umount lustre-MDT0000 complete [12120.466052] LustreError: 351418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804815 with bad export cookie 7804708812380966573 [12120.474255] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12120.478943] LustreError: 351418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12120.974338] Lustre: server umount lustre-MDT0001 complete [12133.152460] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12136.003876] Key type lgssc unregistered [12136.367102] LNet: 353575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12136.379407] LNetError: 353575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12136.401440] LNet: Removed LNI 192.168.204.138@tcp [12137.419758] Key type .llcrypt unregistered [12137.422028] Key type ._llcrypt unregistered [12230.935842] Key type ._llcrypt registered [12230.939554] Key type .llcrypt registered [12231.094969] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12231.650374] Key type .llcrypt unregistered [12231.664562] Key type ._llcrypt unregistered [12305.072712] Key type ._llcrypt registered [12305.076344] Key type .llcrypt registered [12305.218293] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12305.492438] Key type .llcrypt unregistered [12305.497143] Key type ._llcrypt unregistered [12330.545229] Key type ._llcrypt registered [12330.552241] Key type .llcrypt registered [12330.650330] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12330.908489] Key type .llcrypt unregistered [12330.912085] Key type ._llcrypt unregistered [12360.848977] Key type ._llcrypt registered [12360.853455] Key type .llcrypt registered [12360.943749] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12361.245273] Key type .llcrypt unregistered [12361.248990] Key type ._llcrypt unregistered [12436.069803] Key type ._llcrypt registered [12436.073107] Key type .llcrypt registered [12436.173773] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12436.528312] Key type .llcrypt unregistered [12436.534709] Key type ._llcrypt unregistered [12464.655563] Key type ._llcrypt registered [12464.658117] Key type .llcrypt registered [12464.756940] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12465.038810] Key type .llcrypt unregistered [12465.041134] Key type ._llcrypt unregistered [12487.513088] Key type ._llcrypt registered [12487.518245] Key type .llcrypt registered [12487.634543] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12487.873313] Key type .llcrypt unregistered [12487.878503] Key type ._llcrypt unregistered [12513.009130] Key type ._llcrypt registered [12513.013031] Key type .llcrypt registered [12513.095921] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12513.328304] Key type .llcrypt unregistered [12513.334306] Key type ._llcrypt unregistered [12538.476906] Key type ._llcrypt registered [12538.480962] Key type .llcrypt registered [12538.616807] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12538.858982] Key type .llcrypt unregistered [12538.864929] Key type ._llcrypt unregistered [12563.629987] Key type ._llcrypt registered [12563.635929] Key type .llcrypt registered [12563.796634] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12564.093359] Key type .llcrypt unregistered [12564.094971] Key type ._llcrypt unregistered [12616.228347] Key type ._llcrypt registered [12616.231270] Key type .llcrypt registered [12616.331791] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12616.591717] Key type .llcrypt unregistered [12616.595564] Key type ._llcrypt unregistered [12635.387667] Key type ._llcrypt registered [12635.390140] Key type .llcrypt registered [12635.466897] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12635.730799] Key type .llcrypt unregistered [12635.734177] Key type ._llcrypt unregistered [12658.141746] Key type ._llcrypt registered [12658.143405] Key type .llcrypt registered [12658.216591] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12658.476725] Key type .llcrypt unregistered [12658.478976] Key type ._llcrypt unregistered [12695.508742] Key type ._llcrypt registered [12695.512118] Key type .llcrypt registered [12695.601917] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [12707.535507] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [12708.208154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12708.256206] alg: No test for adler32 (adler32-zlib) [12709.219654] Lustre: Lustre: Build Version: 2.17.57_57_g44c575c [12709.382320] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [12711.032192] Key type lgssc registered [12711.839288] Lustre: Echo OBD driver; http://www.lustre.org/ [12736.761097] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12736.780518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.968921] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12737.989351] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12738.047983] Lustre: tst250fs-MDT0000: new disk, initializing [12738.105348] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12738.117730] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12741.392404] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12748.706170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.842657] Lustre: 373738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12748.891629] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12748.895378] Lustre: Skipped 1 previous similar message [12748.969633] Lustre: tst250fs-MDT0001: new disk, initializing [12749.047929] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12749.082342] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12749.091155] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12752.702579] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12756.474727] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12761.183845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12761.419730] Lustre: tst250fs-OST0000: new disk, initializing [12761.428603] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12761.438827] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [12761.439868] Lustre: 375370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12761.505319] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12766.402227] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12766.749985] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12766.757961] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12766.788721] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12774.253423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12774.390230] Lustre: tst250fs-OST0001: new disk, initializing [12774.394309] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12774.399754] Lustre: 376239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12774.481079] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12778.011946] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12778.023084] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12778.109347] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12779.053738] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug -1 all [12783.588500] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12785.620845] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12785.626871] Lustre: Skipped 2 previous similar messages [12800.485520] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12800.493987] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.509225] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12803.561645] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.562034] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12803.575482] Lustre: Skipped 1 previous similar message [12803.577836] Lustre: Skipped 1 previous similar message [12808.680013] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12808.686264] Lustre: Skipped 5 previous similar messages [12813.280402] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12813.405671] Lustre: server umount tst250fs-MDT0000 complete [12815.922845] LustreError: 373731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805511 with bad export cookie 894028531928664229 [12815.924870] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12815.933782] LustreError: 373731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12816.199724] Lustre: server umount tst250fs-MDT0001 complete [12829.395444] Lustre: server umount tst250fs-OST0000 complete [12841.651994] Lustre: server umount tst250fs-OST0001 complete [12856.758678] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [12863.019701] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [12888.212855] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:39:41 (1787805581) [12889.761068] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12903.567891] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [12913.150330] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [12944.391217] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12653 sec ========= 00:40:37 (1787805637) [12946.230367] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:40:39 (1787805639) === [12949.533172] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:40:43 (1787805643) === [12986.414641] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [12988.902822] Key type lgssc unregistered [12989.149400] LNet: 385383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12989.164525] LNetError: 385383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12989.180743] LNet: Removed LNI 192.168.204.138@tcp [12990.080187] Key type .llcrypt unregistered [12990.084127] Key type ._llcrypt unregistered