[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 370261069 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002246] x2apic enabled [ 0.003003] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006385] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009077] LSM: Security Framework initializing [ 0.010036] Yama: becoming mindful. [ 0.010795] SELinux: Initializing. [ 0.011042] *** VALIDATE selinux *** [ 0.018360] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022686] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025081] *** VALIDATE tmpfs *** [ 0.027259] *** VALIDATE proc *** [ 0.028190] *** VALIDATE cgroup *** [ 0.029004] *** VALIDATE cgroup2 *** [ 0.030176] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032018] Spectre V2 : User space: Vulnerable [ 0.033004] Speculative Store Bypass: Vulnerable [ 0.035605] debug: unmapping init [mem 0xffffffff8e059000-0xffffffff8e060fff] [ 0.037185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038527] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039022] ... version: 2 [ 0.040008] ... bit width: 48 [ 0.041015] ... generic registers: 4 [ 0.042008] ... value mask: 0000ffffffffffff [ 0.043007] ... max period: 00007fffffffffff [ 0.044027] ... fixed-purpose events: 3 [ 0.045009] ... event mask: 000000070000000f [ 0.047280] rcu: Hierarchical SRCU implementation. [ 0.049839] smp: Bringing up secondary CPUs ... [ 0.050478] x86: Booting SMP configuration: [ 0.051011] .... node #0, CPUs: #1 #2 #3 [ 0.057108] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.249407] node 0 deferred pages initialised in 186ms [ 0.254195] devtmpfs: initialized [ 0.255285] x86/mm: Memory block size: 128MB [ 0.258181] gcov: version magic: 0x41383552 [ 0.262208] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.266158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269487] pinctrl core: initialized pinctrl subsystem [ 0.271232] [ 0.271735] ************************************************************* [ 0.275011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277009] ** ** [ 0.279009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282013] ** ** [ 0.284011] ** This means that this kernel is built to expose internal ** [ 0.286015] ** IOMMU data structures, which may compromise security on ** [ 0.288009] ** your system. ** [ 0.290015] ** ** [ 0.293014] ** If you see this message and you are not debugging the ** [ 0.295009] ** kernel, report this immediately to your vendor! ** [ 0.297018] ** ** [ 0.300012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302011] ************************************************************* [ 0.304901] NET: Registered protocol family 16 [ 0.307573] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.310052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316108] cpuidle: using governor menu [ 0.319278] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.321711] PCI: Using configuration type 1 for base access [ 0.325205] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336063] cryptd: max_cpu_qlen set to 1000 [ 0.338198] ACPI: Added _OSI(Module Device) [ 0.339019] ACPI: Added _OSI(Processor Device) [ 0.340000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.345477] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.350272] ACPI: Interpreter enabled [ 0.351037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352007] ACPI: Using IOAPIC for interrupt routing [ 0.353071] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.356249] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363601] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377381] acpiphp: Slot [2] registered [ 0.378067] acpiphp: Slot [5] registered [ 0.379063] acpiphp: Slot [6] registered [ 0.381067] acpiphp: Slot [7] registered [ 0.382099] acpiphp: Slot [8] registered [ 0.383049] acpiphp: Slot [9] registered [ 0.384047] acpiphp: Slot [10] registered [ 0.385047] acpiphp: Slot [3] registered [ 0.386010] acpiphp: Slot [4] registered [ 0.387032] acpiphp: Slot [11] registered [ 0.388042] acpiphp: Slot [12] registered [ 0.389029] acpiphp: Slot [13] registered [ 0.389996] acpiphp: Slot [14] registered [ 0.391047] acpiphp: Slot [15] registered [ 0.392028] acpiphp: Slot [16] registered [ 0.393080] acpiphp: Slot [17] registered [ 0.394012] acpiphp: Slot [18] registered [ 0.395047] acpiphp: Slot [19] registered [ 0.396007] acpiphp: Slot [20] registered [ 0.396912] acpiphp: Slot [21] registered [ 0.398051] acpiphp: Slot [22] registered [ 0.399000] acpiphp: Slot [23] registered [ 0.399000] acpiphp: Slot [24] registered [ 0.399071] acpiphp: Slot [25] registered [ 0.400000] acpiphp: Slot [26] registered [ 0.402065] acpiphp: Slot [27] registered [ 0.404071] acpiphp: Slot [28] registered [ 0.406117] acpiphp: Slot [29] registered [ 0.407075] acpiphp: Slot [30] registered [ 0.408062] acpiphp: Slot [31] registered [ 0.409044] PCI host bridge to bus 0000:00 [ 0.410013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.418011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.420012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423891] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426176] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.439639] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.442010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447537] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.450680] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.453027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.455767] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.460014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473871] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.483366] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.517059] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.549025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.559000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.569034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.576029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.604000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.615019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.630026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639484] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.645020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.650015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.669419] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.674022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.679018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.696016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.709889] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.721017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.737017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.762102] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.768559] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774541] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.780167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.785116] iommu: Default domain type: Passthrough [ 0.786000] SCSI subsystem initialized [ 0.787243] ACPI: bus type USB registered [ 0.788090] usbcore: registered new interface driver usbfs [ 0.790146] usbcore: registered new interface driver hub [ 0.792115] usbcore: registered new device driver usb [ 0.794138] pps_core: LinuxPPS API ver. 1 registered [ 0.795007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.798083] PTP clock support registered [ 0.799177] EDAC MC: Ver: 3.0.0 [ 0.801181] PCI: Using ACPI for IRQ routing [ 0.802915] NetLabel: Initializing [ 0.804009] NetLabel: domain hash size = 128 [ 0.805006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.806088] NetLabel: unlabeled traffic allowed by default [ 0.807325] vgaarb: loaded [ 0.809384] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.810014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.820000] clocksource: Switched to clocksource kvm-clock [ 0.935347] VFS: Disk quotas dquot_6.6.0 [ 0.936747] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938978] *** VALIDATE ramfs *** [ 0.940062] *** VALIDATE hugetlbfs *** [ 0.941377] pnp: PnP ACPI init [ 0.943844] pnp: PnP ACPI: found 6 devices [ 0.957825] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.960248] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.961443] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.963621] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966288] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968434] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.971281] NET: Registered protocol family 2 [ 0.973517] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.978180] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.988054] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.991145] TCP: Hash tables configured (established 65536 bind 65536) [ 0.994495] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.997379] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003133] NET: Registered protocol family 1 [ 1.006233] RPC: Registered named UNIX socket transport module. [ 1.009127] RPC: Registered udp transport module. [ 1.015184] RPC: Registered tcp transport module. [ 1.016795] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.018865] NET: Registered protocol family 44 [ 1.020540] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.022420] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.024543] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.027139] PCI: CLS 0 bytes, default 64 [ 1.028721] Unpacking initramfs... [ 2.645882] debug: unmapping init [mem 0xffff9e20fcc54000-0xffff9e20fffbffff] [ 2.652376] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.654649] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.658968] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.190829] Initialise system trusted keyrings [ 3.192287] Key type blacklist registered [ 3.197766] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.209059] zbud: loaded [ 3.211802] *** VALIDATE nfs *** [ 3.213966] *** VALIDATE nfs4 *** [ 3.215265] pstore: using deflate compression [ 3.218849] Platform Keyring initialized [ 3.329724] NET: Registered protocol family 38 [ 3.331338] Key type asymmetric registered [ 3.333470] Asymmetric key parser 'x509' registered [ 3.335347] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.503651] io scheduler mq-deadline registered [ 3.505468] io scheduler kyber registered [ 3.507189] io scheduler bfq registered [ 3.509190] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.512384] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.515367] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.518449] ACPI: Power Button [PWRF] [ 3.524708] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.531916] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.544788] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.551353] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.598164] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.627552] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.656847] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.661600] Non-volatile memory driver v1.3 [ 3.663579] Linux agpgart interface v0.103 [ 3.698720] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.702109] vda: detected capacity change from 0 to 74756096 [ 3.716758] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.719836] vdb: detected capacity change from 0 to 1073741824 [ 3.733310] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.736135] vdc: detected capacity change from 0 to 2621440000 [ 3.756347] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.759140] vdd: detected capacity change from 0 to 2621440000 [ 3.774605] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.777425] vde: detected capacity change from 0 to 4294967296 [ 3.795299] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.798075] vdf: detected capacity change from 0 to 4294967296 [ 3.805087] libphy: Fixed MDIO Bus: probed [ 3.811571] usbcore: registered new interface driver usbserial_generic [ 3.814195] usbserial: USB Serial support registered for generic [ 3.816987] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.821028] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.822917] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.825193] mousedev: PS/2 mouse device common for all mice [ 3.828190] rtc_cmos 00:05: RTC can wake from S4 [ 3.833968] rtc_cmos 00:05: registered as rtc0 [ 3.835384] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.839748] intel_pstate: CPU model not supported [ 3.840730] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.844643] hid: raw HID events driver (C) Jiri Kosina [ 3.849665] usbcore: registered new interface driver usbhid [ 3.851332] usbhid: USB HID core driver [ 3.852748] drop_monitor: Initializing network drop monitor service [ 3.856381] Initializing XFRM netlink socket [ 3.858295] NET: Registered protocol family 10 [ 3.862391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.863311] Segment Routing with IPv6 [ 3.868104] NET: Registered protocol family 17 [ 3.891755] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.893875] mpls_gso: MPLS GSO support [ 3.904045] RAS: Correctable Errors collector initialized. [ 3.906204] AVX version of gcm_enc/dec engaged. [ 3.908188] AES CTR mode by8 optimization enabled [ 3.993563] sched_clock: Marking stable (3993461688, 0)->(4704434660, -710972972) [ 3.997393] registered taskstats version 1 [ 3.999624] Loading compiled-in X.509 certificates [ 4.001599] zswap: loaded using pool lzo/zbud [ 4.025914] Key type big_key registered [ 4.039035] Key type encrypted registered [ 4.040528] ima: No TPM chip found, activating TPM-bypass! [ 4.042403] ima: Allocated hash algorithm: sha1 [ 4.044137] ima: No architecture policies found [ 4.045853] evm: Initialising EVM extended attributes: [ 4.047721] evm: security.selinux [ 4.048933] evm: security.ima [ 4.050091] evm: security.capability [ 4.051317] evm: HMAC attrs: 0x1 [ 4.053784] rtc_cmos 00:05: setting system clock to 2026-08-22 04:26:20 UTC (1787372780) [ 4.060257] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 4.063212] debug: unmapping init [mem 0xffffffff8dd82000-0xffffffff8e058fff] [ 4.069071] Write protecting the kernel read-only data: 28672k [ 4.072554] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 4.075165] debug: unmapping init [mem 0xffffffff8cd14000-0xffffffff8cdfffff] [ 4.112760] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.121378] systemd[1]: Detected virtualization kvm. [ 4.123308] systemd[1]: Detected architecture x86-64. [ 4.125487] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.155679] systemd[1]: No hostname configured. [ 4.157826] systemd[1]: Set hostname to . [ 4.160433] random: systemd: uninitialized urandom read (16 bytes read) [ 4.163670] systemd[1]: Initializing machine ID from random generator. [ 4.255316] random: ln: uninitialized urandom read (6 bytes read) [ 4.516852] random: systemd: uninitialized urandom read (16 bytes read) [ 4.521899] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.531057] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.540862] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.760483] device-mapper: uevent: version 1.0.3 [ 7.762441] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.203850] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook...[ 9.232377] random: fast init done [ 9.472124] scsi host0: ata_piix [ 9.538182] scsi host1: ata_piix [ 9.543752] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.616903] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.715324] random: crng init done [ 15.724785] random: 7 urandom warning(s) missed due to ratelimiting [ 18.246181] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK [0[ 20.652618] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) m] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.047634] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.901601] SELinux: Disabled at runtime. [ 23.980422] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.990433] systemd[1]: Detected virtualization kvm. [ 23.992871] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.363471] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.373494] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.761582] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.780312] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.789768] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.814491] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.834936] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ 27.740515] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.983538] hrtimer: interrupt took 5733479 ns [ 30.070145] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.492513] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.645618] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit)[ 39.708484] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.734687] EDAC sbridge: Ver: 1.1.2 [*** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ *] A start job is running for Configur…only root support (18s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [*** ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (21s / no limit)[ 47.977762] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (21s / no limit) [** ] A start job is running for Configur…only root support (22s / no limit) [*** ] A start job is running for Configur…only root support (22s / no limit)[ 49.074818] NFS: Registering the id_resolver key type [ 49.081914] Key type id_resolver registered [ 49.085262] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (23s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ ***] A start job is running for Rebuild …amic Linker Cache (30s / no limit) [ **] A start job is running for Rebuild …amic Linker Cache (30s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg115-server login: [ 160.141069] libcfs: loading out-of-tree module taints kernel. [ 160.209383] Key type ._llcrypt registered [ 160.213781] Key type .llcrypt registered [ 160.408248] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 180.247712] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 182.149658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 182.165765] alg: No test for adler32 (adler32-zlib) [ 183.892726] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 184.648548] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 186.447184] Key type lgssc registered [ 188.496095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 207.063202] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 251.019540] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 266.066306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 266.131137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.409495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 267.466539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 267.550763] Lustre: lustre-MDT0000: new disk, initializing [ 267.670552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 267.711281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 272.204378] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.013462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.205546] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 285.292744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 285.300629] Lustre: Skipped 1 previous similar message [ 285.498378] Lustre: lustre-MDT0001: new disk, initializing [ 285.606765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.644274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 285.655204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 289.675490] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.282547] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 305.965224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.314270] Lustre: lustre-OST0000: new disk, initializing [ 306.325716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 306.350768] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 306.469176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.699779] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 315.924324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 315.946500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 316.071346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 328.045672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 328.183913] Lustre: lustre-OST0001: new disk, initializing [ 328.187636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 328.193826] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 328.266468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 334.408734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 334.440208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 334.586178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 334.698976] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 346.793865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 355.455268] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 363.719768] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing check_logdir /tmp/testlogs/ [ 369.623619] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing yml_node [ 374.471797] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 377.761124] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 381.365474] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 384.000637] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 00:32:37 EDT 2026 [ 405.785528] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 407.509615] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 423.909441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 423.919664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.932649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.975713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.976294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.984948] Lustre: Skipped 2 previous similar messages [ 427.016745] Lustre: Skipped 2 previous similar messages [ 429.664327] Lustre: server umount lustre-MDT0000 complete [ 432.903844] LustreError: 7474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373209 with bad export cookie 1509187019600844327 [ 432.907613] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 432.924743] LustreError: 7474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 437.221295] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 437.222930] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.232751] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 437.251537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 439.317825] Lustre: server umount lustre-MDT0001 complete [ 442.995251] Lustre: server umount lustre-OST0000 complete [ 446.572314] Lustre: server umount lustre-OST0001 complete [ 453.423311] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 465.482711] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 507.256325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.475229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 507.526717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 507.637952] Lustre: lustre-MDT0000: new disk, initializing [ 507.697932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.723868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 512.977823] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 524.750949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.925638] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 525.010151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 525.013716] Lustre: Skipped 1 previous similar message [ 525.077274] Lustre: lustre-MDT0001: new disk, initializing [ 525.174190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.206445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 525.218062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 530.302536] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 541.013355] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.788083] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 556.252626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.692788] Lustre: lustre-OST0000: new disk, initializing [ 556.703564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 556.710978] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 556.818772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.472101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 558.483866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 558.661828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 563.894452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 573.830670] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 578.809303] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 579.084525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 584.307751] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 584.650796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 589.283573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.287581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.312527] Lustre: Skipped 1 previous similar message [ 589.325435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.348979] Lustre: Skipped 2 previous similar messages [ 594.402748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.055110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 601.245257] Lustre: server umount lustre-OST0000 complete [ 607.206752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.221490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.229974] Lustre: Skipped 1 previous similar message [ 607.245223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.249235] Lustre: Skipped 3 previous similar messages [ 611.136442] Lustre: server umount lustre-MDT0000 complete [ 614.275878] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373390 with bad export cookie 1509187019600845860 [ 614.280206] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.287119] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 614.583261] Lustre: server umount lustre-MDT0001 complete [ 620.422398] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:36:34 (1787373394) [ 639.150135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.560892] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.663617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.429690] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 645.097310] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.213315] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.399137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.342568] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 666.531273] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 672.241579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 674.790342] Lustre: Failing over lustre-MDT0000 [ 674.947959] Lustre: server umount lustre-MDT0000 complete [ 675.297066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.308189] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.319849] Lustre: Skipped 2 previous similar messages [ 675.351677] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 684.512915] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.528162] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 690.595533] Lustre: 3651:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787373451/real 1787373451] req@ffff9e2043cd1500 x1874196390119552/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787373467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 690.645931] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.943161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.899107] LustreError: 20660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.915214] LustreError: 20660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 701.355168] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 705.227457] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 706.539238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 706.550489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 706.599421] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 715.169610] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 720.865986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.881540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.884236] Lustre: Skipped 2 previous similar messages [ 725.629515] Lustre: server umount lustre-MDT0000 complete [ 730.228158] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373506 with bad export cookie 1509187019600847274 [ 730.236646] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.655452] Lustre: server umount lustre-MDT0001 complete [ 739.803627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.159695] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.169037] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 740.215253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.217855] Lustre: Skipped 2 previous similar messages [ 744.782741] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 755.505958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.309272] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 770.685564] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 777.324325] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 779.370788] Lustre: Failing over lustre-MDT0000 [ 779.611517] Lustre: server umount lustre-MDT0000 complete [ 781.408701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.430229] Lustre: Skipped 2 previous similar messages [ 797.152275] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787373557/real 1787373557] req@ffff9e2043b9b800 x1874196390149248/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787373573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 797.181918] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.574585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.395790] LustreError: 23273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.412352] LustreError: 23273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 808.417030] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e217d65e680 x1874196390152192/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 808.783772] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 813.149390] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 814.049144] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 814.066952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 814.072391] Lustre: Skipped 1 previous similar message [ 814.089062] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 823.655239] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 827.538571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.541490] Lustre: Skipped 2 previous similar messages [ 833.850139] Lustre: server umount lustre-MDT0000 complete [ 837.468178] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373613 with bad export cookie 1509187019600848212 [ 837.478066] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.478223] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 837.980223] Lustre: server umount lustre-MDT0001 complete [ 846.421315] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:40:20 (1787373620) [ 863.145343] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 873.488605] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 914.913440] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 926.329529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.554959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 926.589244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 926.661461] Lustre: lustre-MDT0000: new disk, initializing [ 926.715822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.729562] Lustre: Skipped 2 previous similar messages [ 926.759127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 931.382895] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 943.356584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.481702] Lustre: 29643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 943.689078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 948.887558] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 959.127822] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 965.478990] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.051131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.479346] Lustre: 31616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 974.613112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 974.670812] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 974.740111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 974.764737] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 979.639490] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 989.535107] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 995.018917] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1015.777460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.782274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.785349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1015.812016] Lustre: Skipped 3 previous similar messages [ 1015.825989] Lustre: Skipped 3 previous similar messages [ 1020.020990] Lustre: server umount lustre-OST0000 complete [ 1025.504347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1030.656691] Lustre: server umount lustre-MDT0000 complete [ 1031.142197] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.166063] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1033.975528] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373810 with bad export cookie 1509187019600848618 [ 1033.976798] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1033.980578] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1034.256716] Lustre: server umount lustre-MDT0001 complete [ 1044.476540] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1047.177569] Key type lgssc unregistered [ 1047.517597] LNet: 33598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.524435] LNetError: 33598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.544288] LNet: Removed LNI 192.168.201.115@tcp [ 1048.432150] Key type .llcrypt unregistered [ 1048.433968] Key type ._llcrypt unregistered [ 1072.047257] Key type ._llcrypt registered [ 1072.049475] Key type .llcrypt registered [ 1072.145620] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1072.970602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1073.042177] alg: No test for adler32 (adler32-zlib) [ 1074.061780] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1074.237806] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1075.887170] Key type lgssc registered [ 1076.916294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.377614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.401447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.893021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.408454] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1096.164257] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.175669] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1099.525133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.749296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1104.271940] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1111.849599] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1117.738960] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1124.486873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.890730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.939915] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1130.733711] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1131.118248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1137.735769] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1154.011375] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:45:27 (1787373927) [ 1156.144789] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1158.113136] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1165.231554] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:45:39 (1787373939) [ 1174.892255] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:45:48 (1787373948) [ 1175.855984] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1184.955572] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1184.960279] Lustre: Skipped 2 previous similar messages [ 1186.700026] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:46:00 (1787373960) [ 1190.324712] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1190.329959] Lustre: Skipped 3 previous similar messages [ 1207.753322] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1207.759865] Lustre: Skipped 6 previous similar messages [ 1210.116498] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:46:24 (1787373984) [ 1212.947896] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1216.027947] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1223.645492] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:46:38 (1787373998) [ 1225.235877] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1225.246799] Lustre: Skipped 3 previous similar messages [ 1237.763755] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:46:52 (1787374012) [ 1253.857272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.868406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.871865] Lustre: Skipped 1 previous similar message [ 1253.906133] Lustre: Skipped 1 previous similar message [ 1258.992722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.096177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1264.101776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.113669] Lustre: Skipped 2 previous similar messages [ 1264.292365] Lustre: server umount lustre-OST0000 complete [ 1268.708967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.720352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.740555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.344937] LustreError: 35739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.357603] LustreError: 35739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1274.446914] Lustre: server umount lustre-MDT0000 complete [ 1278.188695] LustreError: 36853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374054 with bad export cookie 16274533641440996457 [ 1278.195252] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.199325] LustreError: 36853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.551392] Lustre: server umount lustre-MDT0001 complete [ 1291.482114] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1295.186161] Key type lgssc unregistered [ 1295.556825] LNet: 42876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.562861] LNetError: 42876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.619079] LNet: Removed LNI 192.168.201.115@tcp [ 1297.896886] Key type .llcrypt unregistered [ 1297.899193] Key type ._llcrypt unregistered [ 1317.836936] Key type ._llcrypt registered [ 1317.839284] Key type .llcrypt registered [ 1317.940716] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 1334.754105] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1335.995303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1336.048971] alg: No test for adler32 (adler32-zlib) [ 1337.177423] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1337.405087] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1339.071247] Key type lgssc registered [ 1340.224600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.019326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.050647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.231657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1375.254200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1375.326963] Lustre: lustre-MDT0000: new disk, initializing [ 1375.402664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.417297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1379.616488] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1389.491780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.581087] Lustre: 46031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1389.634222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1389.641368] Lustre: Skipped 1 previous similar message [ 1389.698405] Lustre: lustre-MDT0001: new disk, initializing [ 1389.796627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.847423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1389.868038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1393.354615] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1402.396211] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.118868] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1413.989797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.290427] Lustre: lustre-OST0000: new disk, initializing [ 1414.297527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1414.309713] Lustre: 47998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1414.422118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.664373] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1421.885248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1421.903083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1422.069901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1428.307482] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1433.322645] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1433.576325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1437.365695] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1437.518091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1442.784221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.789338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.800330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.804227] Lustre: Skipped 1 previous similar message [ 1445.455058] Lustre: server umount lustre-OST0000 complete [ 1451.490324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1451.498692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.515887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.518489] Lustre: Skipped 1 previous similar message [ 1453.030687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.032338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.042842] Lustre: Skipped 1 previous similar message [ 1455.321805] Lustre: server umount lustre-MDT0000 complete [ 1459.184078] LustreError: 46025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374235 with bad export cookie 6549959695679038090 [ 1459.185473] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.201797] LustreError: 46025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1459.589449] Lustre: server umount lustre-MDT0001 complete [ 1468.262286] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:50:42 (1787374242) [ 1474.404928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.932475] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.957080] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1475.026744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.470465] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1480.161742] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.179866] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1485.290064] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.526916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.844416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1492.227418] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1498.750899] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1504.083357] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1510.532101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1510.864923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1516.929815] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1523.039896] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1527.878403] LustreError: 52986:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1528.520854] Lustre: Modifying parameter general.timeout=1 in log params [ 1529.707662] Lustre: Modifying parameter general.timeout=3 in log params [ 1529.712725] Lustre: Skipped 1 previous similar message [ 1532.102261] Lustre: Modifying parameter general.timeout=7 in log params [ 1532.104783] Lustre: Skipped 3 previous similar messages [ 1536.163624] Lustre: Modifying parameter general.timeout=13 in log params [ 1536.167361] Lustre: Skipped 5 previous similar messages [ 1544.937671] Lustre: Modifying parameter general.timeout=25 in log params [ 1544.944956] Lustre: Skipped 11 previous similar messages [ 1561.147267] Lustre: Modifying parameter general.timeout=42 in log params [ 1561.150828] Lustre: Skipped 16 previous similar messages [ 1593.194487] Lustre: Modifying parameter general.timeout=79 in log params [ 1593.197992] Lustre: Skipped 36 previous similar messages [ 1645.731784] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:53:39 (1787374419) [ 1649.088442] Lustre: 59932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1650.735977] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1654.214971] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1654.224609] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1659.143653] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1659.147491] Lustre: Skipped 58 previous similar messages [ 1680.709799] Lustre: 61804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1680.716222] Lustre: 61804:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1680.733712] LustreError: 61804:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1684.175922] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:54:18 (1787374458) [ 1697.761933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1697.774876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.792689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.783790] Lustre: server umount lustre-MDT0000 complete [ 1703.905265] LustreError: 51185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.920166] LustreError: 51185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1705.166884] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374481 with bad export cookie 6549959695679038972 [ 1705.172935] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.184387] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1705.445555] Lustre: server umount lustre-MDT0001 complete [ 1719.592896] Lustre: server umount lustre-OST0000 complete [ 1743.319863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.555916] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1743.804653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.457716] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1755.227422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.294110] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1755.322656] Lustre: 64337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1760.069676] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1767.829361] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1773.057826] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1779.478728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1779.667851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1779.900538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.906681] Lustre: Skipped 1 previous similar message [ 1785.567978] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1791.795323] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1793.263671] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1793.270351] Lustre: Skipped 15 previous similar messages [ 1793.301894] Lustre: 66903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1802.720681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1802.730768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.741032] Lustre: Skipped 2 previous similar messages [ 1802.748110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.753493] Lustre: Skipped 3 previous similar messages [ 1806.079867] Lustre: server umount lustre-OST0000 complete [ 1811.947779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1811.965485] LustreError: Skipped 1 previous similar message [ 1811.971067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.994261] Lustre: Skipped 1 previous similar message [ 1812.007841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1812.010365] Lustre: Skipped 2 previous similar messages [ 1815.833472] Lustre: server umount lustre-MDT0000 complete [ 1818.592223] LustreError: 65294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.616663] LustreError: 65294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1819.598490] LustreError: 65089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374596 with bad export cookie 6549959695679041933 [ 1819.620114] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.084578] Lustre: server umount lustre-MDT0001 complete [ 1831.496656] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1834.464995] Key type lgssc unregistered [ 1834.691863] LNet: 68114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.699106] LNetError: 68114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.715248] LNet: Removed LNI 192.168.201.115@tcp [ 1835.547189] Key type .llcrypt unregistered [ 1835.548910] Key type ._llcrypt unregistered [ 1861.954550] Key type ._llcrypt registered [ 1861.957166] Key type .llcrypt registered [ 1862.073332] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1863.559965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1863.686681] alg: No test for adler32 (adler32-zlib) [ 1864.924571] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1865.178521] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1866.807388] Key type lgssc registered [ 1867.830396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.170087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.222541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.684898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.366745] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1888.241497] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.254750] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1893.351079] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.022228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.999836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.984846] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1909.079446] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1916.530610] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.611313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1925.051520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.067817] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1940.389475] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1960.086709] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1971.681963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.691579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.709695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.242902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.268266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.285141] Lustre: Skipped 1 previous similar message [ 1977.003342] Lustre: server umount lustre-MDT0000 complete [ 1979.367090] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.384798] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1980.832215] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374757 with bad export cookie 8861264275270244485 [ 1980.839138] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.850970] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1981.332547] Lustre: server umount lustre-MDT0001 complete [ 1996.178933] Lustre: server umount lustre-OST0000 complete [ 2028.336403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.868317] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2028.918631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2029.250205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.325370] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2042.657501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.747413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2042.784681] Lustre: 74427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2043.194593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.875941] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2057.946244] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.297369] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2072.960914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.303251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2073.591792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.795574] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2091.316611] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2093.709952] Lustre: 77016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2096.308271] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2104.805217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.823562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.843788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.861303] Lustre: Skipped 2 previous similar messages [ 2109.482801] Lustre: server umount lustre-OST0000 complete [ 2115.042463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.053281] LustreError: Skipped 1 previous similar message [ 2115.055678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.068406] Lustre: Skipped 1 previous similar message [ 2115.079352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.088804] Lustre: Skipped 2 previous similar messages [ 2120.673810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.692826] Lustre: Skipped 2 previous similar messages [ 2129.375225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2129.669840] Lustre: server umount lustre-MDT0000 complete [ 2129.905516] LustreError: 74435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.932784] LustreError: 74435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2135.013207] LustreError: 74433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.036253] LustreError: 74433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2135.733832] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374912 with bad export cookie 8861264275270245325 [ 2135.735563] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.754323] LustreError: 74418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2136.208738] Lustre: server umount lustre-MDT0001 complete [ 2148.055842] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2150.790638] Key type lgssc unregistered [ 2151.062112] LNet: 78234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.069225] LNetError: 78234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.102643] LNet: Removed LNI 192.168.201.115@tcp [ 2152.984454] Key type .llcrypt unregistered [ 2152.988234] Key type ._llcrypt unregistered [ 2175.834699] Key type ._llcrypt registered [ 2175.837608] Key type .llcrypt registered [ 2175.932195] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2177.025936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.222595] alg: No test for adler32 (adler32-zlib) [ 2178.347367] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2178.589416] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2180.287240] Key type lgssc registered [ 2181.424910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2189.746852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2189.775206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.354701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.145526] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2201.572391] LustreError: 79647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.596446] LustreError: 79647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2204.531286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.226747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.555849] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2216.353689] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.876282] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2228.733833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.261122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.783657] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2245.097839] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2250.818648] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2251.818758] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2253.505132] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2253.508323] Lustre: Skipped 1 previous similar message [ 2255.564377] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.571753] Lustre: Skipped 1 previous similar message [ 2260.459847] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2260.467498] Lustre: Skipped 4 previous similar messages [ 2268.528592] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2268.535436] Lustre: Skipped 8 previous similar messages [ 2284.867767] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2284.879024] Lustre: Skipped 16 previous similar messages [ 2317.027951] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2317.034495] Lustre: Skipped 34 previous similar messages [ 2354.733362] Lustre: 87754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2358.020780] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:05:31 (1787375131) [ 2359.632561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2361.435343] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:05:35 (1787375135) [ 2368.479970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.490664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.505651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.533778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.539652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.558105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.695593] Lustre: server umount lustre-OST0000 complete [ 2379.239051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2379.256068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.281828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.287245] Lustre: Skipped 1 previous similar message [ 2383.607863] Lustre: server umount lustre-MDT0000 complete [ 2385.898198] LustreError: 80388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.920398] LustreError: 80388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2388.530770] LustreError: 79628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375164 with bad export cookie 5802380101699822936 [ 2388.544246] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.551754] LustreError: 79628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2389.627683] Lustre: server umount lustre-MDT0001 complete [ 2402.179891] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2405.808762] Key type lgssc unregistered [ 2406.213718] LNet: 89009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.219891] LNetError: 89009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.242086] LNet: Removed LNI 192.168.201.115@tcp [ 2407.253493] Key type .llcrypt unregistered [ 2407.258396] Key type ._llcrypt unregistered [ 2412.370737] Key type ._llcrypt registered [ 2412.375524] Key type .llcrypt registered [ 2412.497641] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2417.949025] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules [ 2419.231539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.285592] alg: No test for adler32 (adler32-zlib) [ 2420.602934] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2420.986749] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2422.711191] Key type lgssc registered [ 2424.183944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.419979] LustreError: 90218:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2444.051793] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2447.063902] LustreError: 90218:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2453.269303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.282742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.771453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.929645] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2465.250118] LustreError: 91215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.278744] LustreError: 91215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2469.035963] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:07:22 (1787375242) [ 2470.372307] LustreError: 91216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.213637] LustreError: 91247:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2474.226336] LustreError: 91247:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2474.230984] LustreError: 91247:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2474.606989] Lustre: server umount lustre-MDT0000 complete [ 2489.418831] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2493.022831] Key type lgssc unregistered [ 2493.307678] LNet: 92742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.316424] LNetError: 92742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.336250] LNet: Removed LNI 192.168.201.115@tcp [ 2494.253237] Key type .llcrypt unregistered [ 2494.255936] Key type ._llcrypt unregistered [ 2513.514267] Key type ._llcrypt registered [ 2513.525140] Key type .llcrypt registered [ 2513.637557] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2514.659288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.836571] alg: No test for adler32 (adler32-zlib) [ 2515.991958] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2516.279688] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2517.951487] Key type lgssc registered [ 2519.209740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.957197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2528.978648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.384234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.696776] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2540.516678] LustreError: 93957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.541922] LustreError: 93957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2545.635584] LustreError: 93958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.888364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.343674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.429771] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2562.398541] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2568.209658] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2574.944196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.333153] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2575.641041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.505238] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2590.852506] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2644.617699] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:10:18 (1787375418) [ 2646.254727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2648.536966] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:10:22 (1787375422) [ 2654.693117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.705146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.716815] Lustre: Skipped 2 previous similar messages [ 2654.728063] Lustre: Skipped 2 previous similar messages [ 2659.809075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.820984] Lustre: Skipped 2 previous similar messages [ 2660.103124] Lustre: server umount lustre-MDT0000 complete [ 2663.592531] LustreError: 93940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375440 with bad export cookie 9774082594691982784 [ 2663.600343] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2663.604042] LustreError: 93940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2663.908985] Lustre: server umount lustre-MDT0001 complete [ 2677.535977] Lustre: server umount lustre-OST0000 complete [ 2688.546779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.969710] LustreError: 98276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.117185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.603440] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2694.115994] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.133898] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2699.238502] LustreError: 98276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.101572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.772711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.233346] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2712.599984] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.582755] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.175321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.416429] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2731.430608] LustreError: 98264:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2731.454604] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2731.464245] LustreError: 100359:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2731.477805] LustreError: 100359:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2731.484347] LustreError: 100359:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2731.574458] Lustre: server umount lustre-OST0000 complete [ 2731.578994] LustreError: 100359:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2739.360707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.565346] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2739.573903] LustreError: 98266:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2739.596411] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2739.612569] LustreError: 100827:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2739.623968] LustreError: 100827:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2739.629834] LustreError: 100827:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2739.742957] Lustre: server umount lustre-OST0000 complete [ 2739.745834] LustreError: 100827:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2747.955583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.137376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2748.140359] Lustre: Found index 0 for lustre-OST0000, updating log [ 2748.151880] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2748.255650] Lustre: lustre-OST0000: new disk, initializing [ 2748.259419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2748.267950] Lustre: 101360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2748.462415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.789851] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2758.237645] Lustre: Failing over lustre-OST0000 [ 2758.388168] Lustre: server umount lustre-OST0000 complete [ 2762.208883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.222922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.233875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.807963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.474473] Lustre: server umount lustre-MDT0000 complete [ 2769.952925] LustreError: 98272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.980297] LustreError: 98272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2772.978337] LustreError: 98258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375549 with bad export cookie 9774082594691985528 [ 2772.983778] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.998158] LustreError: 98258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2773.336582] Lustre: server umount lustre-MDT0001 complete [ 2783.088988] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:12:37 (1787375557) [ 2801.942761] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2814.166608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.624179] LustreError: 103705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.788863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.529205] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2828.406172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.815883] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2836.622653] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2842.768164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.224294] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2856.211153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.304758] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2856.371639] Lustre: lustre-OST0001: new disk, initializing [ 2856.375488] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2856.380251] Lustre: 106019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2856.618947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2856.625454] Lustre: Skipped 2 previous similar messages [ 2862.804263] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2865.233222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2865.240899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2865.330781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2870.377941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2876.038329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2880.489731] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2880.496535] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.507859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2880.515309] Lustre: Skipped 2 previous similar messages [ 2886.709968] Lustre: server umount lustre-MDT0001 complete [ 2890.898329] LustreError: 103702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.915514] LustreError: 103702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2895.489181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.581503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2895.592606] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2895.631648] Lustre: 103692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2897.350518] LustreError: 107581:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2897.367076] LustreError: 107581:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2897.389304] LustreError: 103697:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2897.568625] LustreError: 107602:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.115@tcp: cfg command failed: rc = -17 [ 2897.581705] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2897.581705] [ 2900.967517] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2901.510920] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2910.608908] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:14:44 (1787375684) [ 2927.173818] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2944.942762] Lustre: 109637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2968.586943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2972.875968] Lustre: 110772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3059.684940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3059.694954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.704991] Lustre: Skipped 2 previous similar messages [ 3059.710575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3059.716039] Lustre: Skipped 7 previous similar messages [ 3063.298697] Lustre: server umount lustre-MDT0000 complete [ 3065.323684] LustreError: 103700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.353820] LustreError: 103700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3067.361221] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375843 with bad export cookie 9774082594691986522 [ 3067.369936] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.384154] LustreError: 106018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3067.707262] Lustre: server umount lustre-MDT0001 complete [ 3082.179357] Lustre: server umount lustre-OST0000 complete [ 3085.803627] Lustre: 93271:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375846/real 1787375846] req@ffff9e2174c62a00 x1874198834532608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3085.852906] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.865841] Lustre: Skipped 3 previous similar messages [ 3086.503832] Lustre: server umount lustre-OST0001 complete [ 3092.524290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3102.623550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3113.381623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3125.637873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3146.958409] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3158.502280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.519649] Lustre: lustre-MDT0000: reset Object Index mappings [ 3159.094996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.104201] Lustre: Skipped 1 previous similar message [ 3164.200284] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3173.858072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.893500] Lustre: lustre-MDT0001: reset Object Index mappings [ 3179.596203] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3183.180440] Lustre: 115868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3191.062817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3199.205508] LustreError: 116222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.223842] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3199.225420] LustreError: 116222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3199.490832] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3204.351568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3209.310362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3215.368901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3215.380131] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3217.561287] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3226.755543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3231.097889] Lustre: 117745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3283.840130] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:20:58 (1787376058) [ 3312.610111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.619216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.633835] Lustre: Skipped 2 previous similar messages [ 3312.647080] Lustre: Skipped 7 previous similar messages [ 3323.359212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3323.630814] Lustre: server umount lustre-MDT0000 complete [ 3327.380525] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376103 with bad export cookie 9774082594692033394 [ 3327.392801] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3327.399041] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.127521] Lustre: server umount lustre-MDT0001 complete [ 3344.677583] Lustre: server umount lustre-OST0000 complete [ 3355.562457] Lustre: server umount lustre-OST0001 complete [ 3363.707338] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 3378.608502] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3422.328392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.697940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3422.748666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3422.890036] Lustre: lustre-MDT0000: new disk, initializing [ 3423.062602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.068311] Lustre: Skipped 3 previous similar messages [ 3423.105725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3428.657837] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3436.499276] Lustre: server umount lustre-MDT0000 complete [ 3448.959520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.307400] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3449.339763] Lustre: 122040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3449.349022] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3455.294688] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3461.764304] Lustre: server umount lustre-MDT0000 complete [ 3472.970770] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:24:06 (1787376246) [ 3474.970566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3477.546796] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:24:11 (1787376251) [ 3499.985639] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3512.302959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.823275] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3529.151095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.354540] Lustre: 124093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3529.441888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3529.449557] Lustre: Skipped 1 previous similar message [ 3529.588437] Lustre: lustre-MDT0001: new disk, initializing [ 3529.785564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3529.801881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3535.078183] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3540.627538] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3548.024895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3548.432392] Lustre: lustre-OST0000: new disk, initializing [ 3548.440687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3548.459407] Lustre: 125671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3550.254958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3550.268909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3550.396556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3555.611860] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3567.341103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.597485] Lustre: lustre-OST0001: new disk, initializing [ 3567.603805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3567.612088] Lustre: 126541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3569.665379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3569.672428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3569.787413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3575.568851] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3587.733882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3593.047814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4010.982865] Lustre: 132960:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4010.989549] Lustre: 132960:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4010.994304] Lustre: 132960:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4010.999711] Lustre: 132960:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4011.006290] Lustre: 132960:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4011.009501] Lustre: 132960:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4252.223777] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:37:06 (1787377026) [ 4266.983452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.987683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.997977] Lustre: Skipped 3 previous similar messages [ 4267.013201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4267.016926] Lustre: Skipped 12 previous similar messages [ 4273.053232] Lustre: server umount lustre-MDT0000 complete [ 4274.147044] LustreError: 125665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.183495] LustreError: 125665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4276.369502] LustreError: 132973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377052 with bad export cookie 9774082594692101077 [ 4276.376608] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.385892] LustreError: 132973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4279.266196] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.296575] Lustre: Skipped 3 previous similar messages [ 4279.307301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4279.317842] Lustre: Skipped 5 previous similar messages [ 4282.950385] Lustre: server umount lustre-MDT0001 complete [ 4292.377151] Lustre: server umount lustre-OST0000 complete [ 4303.059444] Lustre: server umount lustre-OST0001 complete [ 4309.859349] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4319.782353] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4353.618891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.798564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4353.850508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.958360] Lustre: lustre-MDT0000: new disk, initializing [ 4354.034855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.041551] Lustre: Skipped 5 previous similar messages [ 4354.067440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4358.137388] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4368.181610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.306692] Lustre: 136008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.359718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4368.362516] Lustre: Skipped 1 previous similar message [ 4368.410862] Lustre: lustre-MDT0001: new disk, initializing [ 4368.522773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4368.534273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4372.591709] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4381.608409] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4387.230773] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4393.630175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4393.849618] Lustre: lustre-OST0000: new disk, initializing [ 4393.853376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4393.860948] Lustre: 137974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4393.947088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.957608] Lustre: Skipped 1 previous similar message [ 4395.573201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4395.585215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4395.784849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4400.520624] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4409.941060] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4419.882939] Lustre: lustre-MDD0000: changelog on [ 4422.583314] Lustre: lustre-MDD0001: changelog on [ 6331.811366] Lustre: lustre-MDD0001: changelog off [ 6333.601620] Lustre: 136016:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6334.307226] Lustre: lustre-MDD0000: changelog off [ 6341.094516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6341.117842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.134164] Lustre: Skipped 1 previous similar message [ 6341.138928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6341.150461] Lustre: Skipped 2 previous similar messages [ 6341.619228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6346.705566] Lustre: server umount lustre-OST0000 complete [ 6351.842334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.846548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6351.857625] Lustre: Skipped 2 previous similar messages [ 6351.861413] Lustre: Skipped 1 previous similar message [ 6356.902363] Lustre: server umount lustre-MDT0000 complete [ 6356.966280] LustreError: 136017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.988441] LustreError: 136017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6360.708860] LustreError: 136001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379137 with bad export cookie 9774082594692879456 [ 6360.736549] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6361.064882] Lustre: server umount lustre-MDT0001 complete [ 6373.810423] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6376.913275] Key type lgssc unregistered [ 6377.172926] LNet: 147127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6377.179430] LNetError: 147127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.213296] LNet: Removed LNI 192.168.201.115@tcp [ 6379.327864] Key type .llcrypt unregistered [ 6379.329642] Key type ._llcrypt unregistered [ 6400.630574] Key type ._llcrypt registered [ 6400.633393] Key type .llcrypt registered [ 6400.732176] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 6419.409710] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6420.961400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6421.023662] alg: No test for adler32 (adler32-zlib) [ 6422.442671] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 6422.761646] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6424.599189] Key type lgssc registered [ 6426.747237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6468.309671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6468.339654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.683351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6469.760257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6469.858833] Lustre: lustre-MDT0000: new disk, initializing [ 6469.966926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.981272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6476.264592] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6487.744660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.871038] Lustre: 150328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6487.927505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6487.932689] Lustre: Skipped 1 previous similar message [ 6488.002864] Lustre: lustre-MDT0001: new disk, initializing [ 6488.099916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.143942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6488.163649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6492.929752] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6503.440255] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6510.372665] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6519.415347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.907651] Lustre: lustre-OST0000: new disk, initializing [ 6519.915717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6519.928946] Lustre: 152296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6520.103651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6526.083692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6526.088731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6526.234990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6526.253965] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6536.374721] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.931996] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_module obdecho/obdecho [ 6545.949980] LustreError: 153301:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6545.962715] LustreError: 153301:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6547.835520] LustreError: 153397:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6562.277701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.290499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.301647] Lustre: Skipped 2 previous similar messages [ 6562.327737] Lustre: Skipped 2 previous similar messages [ 6567.394583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.403591] Lustre: Skipped 2 previous similar messages [ 6568.131844] Lustre: server umount lustre-MDT0000 complete [ 6571.775252] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379348 with bad export cookie 8281268277489869870 [ 6571.788687] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.798086] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6572.056758] Lustre: server umount lustre-MDT0001 complete [ 6586.202099] Lustre: server umount lustre-OST0000 complete [ 6612.300484] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 6627.262917] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6669.120494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.463266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6669.502568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6669.600971] Lustre: lustre-MDT0000: new disk, initializing [ 6669.726739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.758511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6674.256444] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6685.714104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.918922] Lustre: 157139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6686.011807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6686.018368] Lustre: Skipped 1 previous similar message [ 6686.158298] Lustre: lustre-MDT0001: new disk, initializing [ 6686.313828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6686.388499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6686.421891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6691.971408] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6702.430860] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.259874] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.127900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.453474] Lustre: lustre-OST0000: new disk, initializing [ 6717.457039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6717.465725] Lustre: 159108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6717.540903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6723.990418] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6724.721269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6724.735577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6724.870825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6733.285542] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6738.004849] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6738.189290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6743.455303] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6743.642871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6745.568495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6745.571347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.577096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.599771] Lustre: Skipped 1 previous similar message [ 6750.691315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.699029] Lustre: Skipped 2 previous similar messages [ 6751.712533] Lustre: server umount lustre-OST0000 complete [ 6757.856837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6757.865816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.884193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.931430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.270174] Lustre: server umount lustre-MDT0000 complete [ 6766.453334] LustreError: 157129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379542 with bad export cookie 8281268277489870836 [ 6766.455101] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6766.472247] LustreError: 157129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6766.859659] Lustre: server umount lustre-MDT0001 complete [ 6770.861988] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:19:04 (1787379544) [ 6773.142409] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6775.373584] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:19:09 (1787379549) [ 6784.323515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.835727] LustreError: 161518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.871723] LustreError: 161518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6784.949529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6790.120239] LustreError: 161519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6790.395807] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6795.239200] LustreError: 161518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.111191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.353054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6800.363074] Lustre: Skipped 2 previous similar messages [ 6805.673391] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6813.712282] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6820.243899] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.379218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.900446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.909519] Lustre: Skipped 1 previous similar message [ 6835.184290] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6843.788073] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6855.661291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.676711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.683288] Lustre: Skipped 1 previous similar message [ 6856.562210] Lustre: server umount lustre-MDT0000 complete [ 6859.967980] LustreError: 161515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.075192] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379636 with bad export cookie 8281268277489871718 [ 6860.080045] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.102707] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6865.035229] LustreError: 161513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.061274] LustreError: 161513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6866.802991] Lustre: server umount lustre-MDT0001 complete [ 6873.515677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6888.232448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.583243] LustreError: 165359:0:(llog.c:1655:llog_backup()) MGC192.168.201.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6888.592968] Lustre: 165359:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6889.677137] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.699934] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6905.993954] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.014026] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6940.642987] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.674950] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6949.343538] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6949.353599] Lustre: 165359:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6990.321387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6995.056483] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6995.454343] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7005.656352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.669785] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.685299] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7021.032206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7032.288372] LustreError: 148549:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a536869 x1874202931108480/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7037.449890] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7037.935031] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7046.867943] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7048.766968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7056.566528] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7058.152819] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7078.887112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7078.890826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.901771] Lustre: Skipped 3 previous similar messages [ 7078.950309] Lustre: server umount lustre-MDT0000 complete [ 7083.858734] LustreError: 164301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379860 with bad export cookie 8281268277489878207 [ 7083.861562] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7083.885980] LustreError: 164301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7084.006323] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7084.009934] Lustre: Skipped 3 previous similar messages [ 7084.386693] Lustre: server umount lustre-MDT0001 complete [ 7089.640286] Lustre: server umount lustre-OST0000 complete [ 7113.691792] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 7126.451752] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7170.551667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.921719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7170.960305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7171.091922] Lustre: lustre-MDT0000: new disk, initializing [ 7171.156730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.165139] Lustre: Skipped 1 previous similar message [ 7171.183920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7175.912665] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7188.119741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.200793] Lustre: 170470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7188.473520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7194.164520] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7207.370114] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7216.850214] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7225.993963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.204535] Lustre: lustre-OST0000: new disk, initializing [ 7226.210845] Lustre: Skipped 1 previous similar message [ 7226.216282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7226.221869] Lustre: Skipped 2 previous similar messages [ 7226.227541] Lustre: 172438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7231.511355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7231.513974] Lustre: Skipped 1 previous similar message [ 7231.516024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7231.601798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7234.854447] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7247.774453] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.954427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7253.130712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7258.921211] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7259.230292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7262.182860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.202549] Lustre: Skipped 3 previous similar messages [ 7262.235622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.249587] Lustre: Skipped 1 previous similar message [ 7268.053468] Lustre: server umount lustre-OST0000 complete [ 7278.633864] Lustre: server umount lustre-MDT0000 complete [ 7282.656460] LustreError: 170481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.674682] LustreError: 170481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 7282.869432] LustreError: 170462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380059 with bad export cookie 8281268277489895014 [ 7282.870287] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.880222] LustreError: 170462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7283.287453] Lustre: server umount lustre-MDT0001 complete [ 7294.361342] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:27:48 (1787380068) [ 7302.464071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.661641] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7319.450690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.720781] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7336.117798] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7342.588553] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7350.325657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.456210] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7368.398294] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.282427] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:29:21 (1787380161) [ 7393.765156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.784427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7393.790861] Lustre: Skipped 3 previous similar messages [ 7393.791679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7393.791685] Lustre: Skipped 6 previous similar messages [ 7398.978337] Lustre: server umount lustre-OST0000 complete [ 7409.904580] Lustre: server umount lustre-MDT0000 complete [ 7414.114161] LustreError: 174836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380190 with bad export cookie 8281268277489895896 [ 7414.136104] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.515449] Lustre: server umount lustre-MDT0001 complete [ 7428.767512] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7432.600767] Key type lgssc unregistered [ 7432.937643] LNet: 178614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7432.945841] LNetError: 178614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7432.958750] LNet: Removed LNI 192.168.201.115@tcp [ 7433.779649] Key type .llcrypt unregistered [ 7433.783205] Key type ._llcrypt unregistered [ 7452.938965] Key type ._llcrypt registered [ 7452.941536] Key type .llcrypt registered [ 7453.045053] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7454.424482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7454.511382] alg: No test for adler32 (adler32-zlib) [ 7455.602477] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7455.891686] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7457.607565] Key type lgssc registered [ 7459.123623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7469.060416] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7469.077449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.703798] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7489.889961] LustreError: 179858:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 7495.007435] LustreError: 179858:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7495.021140] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7495.030061] LustreError: MGC192.168.201.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7495.041828] LustreError: 179858:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7495.056736] LustreError: 179858:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7495.064558] LustreError: 179858:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7495.159482] Lustre: server umount lustre-OST0000 complete [ 7495.161596] LustreError: 179858:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7503.844387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.201034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7509.736746] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7514.604928] LustreError: 180358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.620341] LustreError: 180358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7518.582970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.065354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7523.449184] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7530.646983] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7537.200333] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7544.229681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.400837] LustreError: 182174:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7544.411348] Lustre: 182174:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7544.555486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.905617] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7557.962799] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.160398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.167308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.174511] Lustre: Skipped 2 previous similar messages [ 7560.184448] Lustre: Skipped 2 previous similar messages [ 7564.321982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7564.335621] Lustre: Skipped 2 previous similar messages [ 7566.463490] Lustre: server umount lustre-MDT0000 complete [ 7569.376542] LustreError: 181397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.396094] LustreError: 181397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7570.180584] LustreError: 181091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380346 with bad export cookie 11688363679788711559 [ 7570.190897] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7570.193496] LustreError: 181091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7570.487841] LustreError: 181085:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000051caa3d6 x1874204014178560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7570.672655] Lustre: server umount lustre-MDT0001 complete [ 7586.016967] Lustre: server umount lustre-OST0000 complete [ 7596.181702] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 7606.448696] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7647.121830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.416027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7647.444876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7647.511587] Lustre: lustre-MDT0000: new disk, initializing [ 7647.590966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.611826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7652.464501] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7663.140458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.345848] Lustre: 185697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7663.412628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7663.416422] Lustre: Skipped 1 previous similar message [ 7663.507771] Lustre: lustre-MDT0001: new disk, initializing [ 7663.598514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.643296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7663.661236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7668.374785] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7678.970751] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7685.792388] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7693.300212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7693.611412] Lustre: lustre-OST0000: new disk, initializing [ 7693.614976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7693.630152] Lustre: 187666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7693.738654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.032875] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7701.074978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7701.084813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7701.231792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7713.881923] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7720.538433] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7720.784246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7726.139602] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7726.382975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7732.210466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.214526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.217915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.266207] Lustre: Skipped 1 previous similar message [ 7735.262508] Lustre: server umount lustre-OST0000 complete [ 7740.384710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.393514] LustreError: Skipped 1 previous similar message [ 7740.398777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.424210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.432866] Lustre: Skipped 1 previous similar message [ 7742.438278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.714449] Lustre: server umount lustre-MDT0000 complete [ 7747.553033] LustreError: 188024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.580708] LustreError: 188024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7750.534049] LustreError: 185690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380526 with bad export cookie 11688363679788712280 [ 7750.552640] LustreError: 185690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7750.553095] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7750.919883] Lustre: server umount lustre-MDT0001 complete [ 7761.813574] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:35:35 (1787380535) [ 7783.789608] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7786.917920] Key type lgssc unregistered [ 7787.279955] LNet: 190618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7787.290737] LNetError: 190618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7787.304252] LNet: Removed LNI 192.168.201.115@tcp [ 7788.246316] Key type .llcrypt unregistered [ 7788.250391] Key type ._llcrypt unregistered [ 7809.317728] Key type ._llcrypt registered [ 7809.320172] Key type .llcrypt registered [ 7809.422535] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7811.054271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7811.144341] alg: No test for adler32 (adler32-zlib) [ 7812.251569] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7812.509171] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7814.223222] Key type lgssc registered [ 7815.533695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7823.345751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7823.363986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.798757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.569801] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7835.111962] LustreError: 191838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.134243] LustreError: 191838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7837.755293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.043568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.641524] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7851.464649] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7857.754619] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7866.149284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7866.435434] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7866.447762] LustreError: 193701:0:(llog.c:1696:llog_backup()) MGC192.168.201.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7866.466619] Lustre: 193701:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7866.647920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.259100] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7882.055728] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7888.354165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.373622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.382104] Lustre: Skipped 2 previous similar messages [ 7888.405069] Lustre: Skipped 2 previous similar messages [ 7891.119722] Lustre: server umount lustre-MDT0000 complete [ 7893.476853] LustreError: 193856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7893.509368] LustreError: 193856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7895.646409] LustreError: 191819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380672 with bad export cookie 12546039571891739781 [ 7895.673810] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7896.134994] Lustre: server umount lustre-MDT0001 complete [ 7911.202790] Lustre: server umount lustre-OST0000 complete [ 7925.310207] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 7938.547708] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7988.098410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.572217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7988.620771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7988.792249] Lustre: lustre-MDT0000: new disk, initializing [ 7989.012206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.066167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7994.233453] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8006.040362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.146518] Lustre: 197232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8006.172771] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8006.175246] Lustre: Skipped 1 previous similar message [ 8006.177759] LustreError: 198015:0:(llog.c:1696:llog_backup()) MGC192.168.201.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8006.185442] LustreError: 198015:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8006.188881] Lustre: 198015:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8006.201795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8006.205527] Lustre: Skipped 1 previous similar message [ 8006.274173] Lustre: lustre-MDT0001: new disk, initializing [ 8006.356180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.401053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8006.416568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8012.083278] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8021.798506] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8028.938393] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8037.874350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.166502] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8038.171280] Lustre: Skipped 1 previous similar message [ 8038.174607] LustreError: 199179:0:(llog.c:1696:llog_backup()) MGC192.168.201.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8038.185749] LustreError: 199179:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8038.190011] Lustre: 199179:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8038.209645] Lustre: 199179:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8038.239645] Lustre: lustre-OST0000: new disk, initializing [ 8038.243343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8038.249820] Lustre: 199202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8038.352841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.759858] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8047.162350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8047.169312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8047.240526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8056.567568] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8062.945988] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8063.163967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8069.760561] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8070.081792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8072.675065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.682628] Lustre: Skipped 1 previous similar message [ 8072.685727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.793225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.809242] Lustre: Skipped 2 previous similar messages [ 8078.960786] Lustre: server umount lustre-OST0000 complete [ 8083.424894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8083.427536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.434387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8088.042548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.051603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.613394] Lustre: server umount lustre-MDT0000 complete [ 8093.154548] LustreError: 197243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.173768] LustreError: 197243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8093.625832] LustreError: 197223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380870 with bad export cookie 12546039571891740502 [ 8093.635618] LustreError: 197223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8093.649676] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.213652] Lustre: server umount lustre-MDT0001 complete [ 8102.965899] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:41:17 (1787380877) [ 8120.929626] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8131.623974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.178768] LustreError: 202331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.291910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.017504] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8137.697986] LustreError: 202332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.919171] LustreError: 202332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.950496] LustreError: 202332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8147.367829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.022374] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8156.661700] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8164.806123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8165.223233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.236856] Lustre: Skipped 1 previous similar message [ 8172.419951] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8182.700835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.939977] Lustre: lustre-OST0001: new disk, initializing [ 8182.946647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8182.951668] Lustre: 204638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8188.588910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8188.604520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8188.794908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8191.392104] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8204.356606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8211.877834] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8214.687972] Lustre: DEBUG MARKER: ADD OST3 [ 8220.984484] loop: module loaded [ 8221.073794] loop0: detected capacity change from 0 to 204800000 [ 8221.392118] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8221.409484] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8221.427035] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8228.592314] loop0: detected capacity change from 0 to 204800000 [ 8229.211557] Lustre: DEBUG MARKER: STOP OST3 [ 8235.441345] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.679214] Lustre: lustre-OST0003: new disk, initializing [ 8235.681681] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8235.689700] Lustre: 206671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8235.829590] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8235.843668] Lustre: Skipped 1 previous similar message [ 8236.897911] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8237.092049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8237.098832] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8237.133840] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8238.945100] LustreError: 206778:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8239.000547] Lustre: Failing over lustre-OST0003 [ 8239.243176] Lustre: server umount lustre-OST0003 complete [ 8242.145566] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.155071] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.185574] LustreError: 203817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.477493] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8252.794880] Lustre: DEBUG MARKER: START OST3 again [ 8261.524913] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.094686] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8262.811370] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8264.016500] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8264.019399] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8268.799482] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8293.134066] Lustre: Failing over lustre-OST0003 [ 8293.278434] Lustre: server umount lustre-OST0003 complete [ 8294.880882] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8294.883856] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.890191] LustreError: Skipped 1 previous similar message [ 8294.895835] LustreError: 203819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.895845] LustreError: 203819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8294.967053] Lustre: Skipped 2 previous similar messages [ 8313.313291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8313.319496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.333176] Lustre: Skipped 2 previous similar messages [ 8313.346897] Lustre: Skipped 1 previous similar message [ 8319.583511] Lustre: server umount lustre-MDT0000 complete [ 8323.572374] LustreError: 202311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381100 with bad export cookie 12546039571891741377 [ 8323.608967] LustreError: 202311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8323.617076] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8330.305989] Lustre: server umount lustre-MDT0001 complete [ 8335.069859] Lustre: server umount lustre-OST0000 complete [ 8339.074563] Lustre: server umount lustre-OST0001 complete [ 8347.658526] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8358.835636] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8403.305579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.576438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8403.600870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8403.765994] Lustre: lustre-MDT0000: new disk, initializing [ 8403.903417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8403.916170] Lustre: Skipped 1 previous similar message [ 8403.940106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8409.050203] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8420.992814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.104958] Lustre: 212100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8421.204869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8421.212837] Lustre: Skipped 1 previous similar message [ 8421.302392] Lustre: lustre-MDT0001: new disk, initializing [ 8421.489456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8421.500366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8427.040991] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8438.530376] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8446.735909] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8454.012909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.408779] Lustre: lustre-OST0000: new disk, initializing [ 8454.413143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8454.420899] Lustre: 214073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8456.397696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8456.419738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8456.590089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8461.497743] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8472.185475] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8477.568200] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8477.751705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8483.701210] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8484.019166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8487.399821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8487.419162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.430432] Lustre: Skipped 3 previous similar messages [ 8487.440097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8487.452319] Lustre: Skipped 7 previous similar messages [ 8492.604119] Lustre: server umount lustre-OST0000 complete [ 8503.430630] Lustre: server umount lustre-MDT0000 complete [ 8506.860445] LustreError: 212108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.885146] LustreError: 212108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8507.682846] LustreError: 212093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381284 with bad export cookie 12546039571891743673 [ 8507.691910] LustreError: 212093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8507.693703] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.325559] Lustre: server umount lustre-MDT0001 complete [ 8512.215269] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:48:06 (1787381286) [ 8530.626525] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8546.916142] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8592.225740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.672266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8592.710339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8592.796255] Lustre: lustre-MDT0000: new disk, initializing [ 8592.894338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8597.961759] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8609.380261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8609.584262] Lustre: 218703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8610.020776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8615.777909] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8627.556562] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8635.731831] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8644.448966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.731544] Lustre: 220674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8645.892603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8646.126376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8651.712248] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8663.256548] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8668.735483] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8668.927835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8673.644251] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8673.940290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8676.838857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.839067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.845720] LustreError: Skipped 2 previous similar messages [ 8676.872623] Lustre: Skipped 3 previous similar messages [ 8676.877102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.884610] Lustre: Skipped 4 previous similar messages [ 8682.335772] Lustre: server umount lustre-OST0000 complete [ 8686.578752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8692.899927] Lustre: server umount lustre-MDT0000 complete [ 8696.744696] LustreError: 219500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381473 with bad export cookie 12546039571891744555 [ 8696.746409] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8696.754377] LustreError: 219500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8697.030726] Lustre: server umount lustre-MDT0001 complete [ 8706.459330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.922599] LustreError: 222889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8707.049477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.061463] Lustre: Skipped 5 previous similar messages [ 8713.256652] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8724.514353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.699735] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8742.016040] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8750.707533] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8759.951526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.264264] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8777.218358] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8787.427827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8787.434120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8787.447115] Lustre: Skipped 8 previous similar messages [ 8793.545110] Lustre: server umount lustre-OST0000 complete [ 8804.111441] Lustre: server umount lustre-MDT0000 complete [ 8808.387521] LustreError: 222869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381584 with bad export cookie 12546039571891745437 [ 8808.401789] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8808.416407] LustreError: 222869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8809.189556] Lustre: server umount lustre-MDT0001 complete [ 8918.713754] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8921.974923] Key type lgssc unregistered [ 8922.250789] LNet: 226560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8922.261024] LNetError: 226560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8922.291546] LNet: Removed LNI 192.168.201.115@tcp [ 8923.340290] Key type .llcrypt unregistered [ 8923.345278] Key type ._llcrypt unregistered [ 8950.099631] Key type ._llcrypt registered [ 8950.102226] Key type .llcrypt registered [ 8950.227610] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8965.726816] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8967.178054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8967.220366] alg: No test for adler32 (adler32-zlib) [ 8968.444257] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8968.809809] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8970.570144] Key type lgssc registered [ 8971.820319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9011.462821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9011.497536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.807474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9012.831149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9012.923042] Lustre: lustre-MDT0000: new disk, initializing [ 9013.019230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.047158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9017.983197] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9028.831642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.943943] Lustre: 229917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9028.992359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9028.995379] Lustre: Skipped 1 previous similar message [ 9029.039146] Lustre: lustre-MDT0001: new disk, initializing [ 9029.112965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9029.151613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9029.160385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9033.188087] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9042.130801] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9049.182690] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9055.612050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9055.785826] Lustre: lustre-OST0000: new disk, initializing [ 9055.788575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9055.792889] Lustre: 231885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9055.868132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9061.458230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9061.477244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9061.573373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9061.739808] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9072.374441] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9078.302515] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9078.514893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9083.246220] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9083.472405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9087.457357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.459321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.474451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9087.494030] Lustre: Skipped 1 previous similar message [ 9092.017683] Lustre: server umount lustre-OST0000 complete [ 9095.651892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9095.662586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.675194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9095.679603] Lustre: Skipped 1 previous similar message [ 9097.696604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.698990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9097.711270] Lustre: Skipped 1 previous similar message [ 9101.557309] Lustre: server umount lustre-MDT0000 complete [ 9105.178760] LustreError: 229907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381881 with bad export cookie 14493232437034431020 [ 9105.192875] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9105.195777] LustreError: 229907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9105.718653] Lustre: server umount lustre-MDT0001 complete [ 9127.183362] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9138.200388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9138.505305] LustreError: 234819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.527704] LustreError: 234819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9138.578665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.102268] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9143.789461] LustreError: 234820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9147.888223] LustreError: 234819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.012322] LustreError: 234820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.438510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.880784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.194720] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9165.132632] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9171.872525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9172.200898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.887741] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9186.601604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.799117] Lustre: lustre-OST0001: new disk, initializing [ 9186.804602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9186.811160] Lustre: 237128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9192.023154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9192.032257] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9192.169728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9192.781465] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9203.864846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9210.789297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9223.214670] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:59:57 (1787381997) [ 9226.721674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9226.732536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.747751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.327046] Lustre: server umount lustre-MDT0000 complete [ 9233.377490] LustreError: 234820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.386160] LustreError: 234820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9235.149735] LustreError: 234800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382011 with bad export cookie 14493232437034431909 [ 9235.153209] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9235.164124] LustreError: 234800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9235.480950] Lustre: server umount lustre-MDT0001 complete [ 9248.993708] Lustre: server umount lustre-OST0000 complete [ 9262.700942] Lustre: server umount lustre-OST0001 complete [ 9269.143181] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 9279.557417] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9316.133309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.326991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9316.371807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9316.441366] Lustre: lustre-MDT0000: new disk, initializing [ 9316.532680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.536064] Lustre: Skipped 1 previous similar message [ 9316.551968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9321.338115] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9332.163393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.288078] Lustre: 241085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9332.358657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9332.364562] Lustre: Skipped 1 previous similar message [ 9332.465383] Lustre: lustre-MDT0001: new disk, initializing [ 9332.600613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9332.611428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9337.290274] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9347.424748] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9353.367403] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9359.904296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9360.208483] Lustre: lustre-OST0000: new disk, initializing [ 9360.212499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9360.219072] Lustre: 243054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9360.349478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.355922] Lustre: Skipped 1 previous similar message [ 9361.961486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9361.968963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9362.076283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9366.944453] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9375.752788] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9380.650681] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9380.929573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9385.692857] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9386.097121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9392.631554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9392.635059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.653827] Lustre: Skipped 3 previous similar messages [ 9392.664411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9392.674594] Lustre: Skipped 4 previous similar messages [ 9402.335793] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9402.710418] Lustre: server umount lustre-OST0000 complete [ 9408.495142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.504379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9408.513062] Lustre: Skipped 2 previous similar messages [ 9408.517773] Lustre: Skipped 4 previous similar messages [ 9421.279139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9421.517826] Lustre: server umount lustre-MDT0000 complete [ 9423.842473] LustreError: 241090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9423.859429] LustreError: 241090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9425.559239] LustreError: 241077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382202 with bad export cookie 14493232437034434044 [ 9425.585161] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9426.015825] Lustre: server umount lustre-MDT0001 complete [ 9435.791820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.356655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.233869] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9441.764500] LustreError: 245277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.780862] LustreError: 245277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9448.758715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.475809] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9460.935808] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9466.178737] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9473.456555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.837169] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9489.289064] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9638.893378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9638.906026] LustreError: Skipped 1 previous similar message [ 9638.908202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9638.923859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9638.928292] Lustre: Skipped 5 previous similar messages [ 9649.119339] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9649.316772] Lustre: server umount lustre-OST0000 complete [ 9653.728706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9653.747564] LustreError: Skipped 1 previous similar message [ 9667.040536] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9667.233838] Lustre: server umount lustre-MDT0000 complete [ 9670.634273] LustreError: 246718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.669576] LustreError: 246718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9670.964027] LustreError: 246011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382447 with bad export cookie 14493232437034434926 [ 9670.967368] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.973942] LustreError: 246011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9671.290846] Lustre: server umount lustre-MDT0001 complete [ 9758.408911] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9761.150993] Key type lgssc unregistered [ 9761.489936] LNet: 248887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9761.505398] LNetError: 248887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9761.523963] LNet: Removed LNI 192.168.201.115@tcp [ 9762.542170] Key type .llcrypt unregistered [ 9762.544519] Key type ._llcrypt unregistered [ 9789.048830] Key type ._llcrypt registered [ 9789.050478] Key type .llcrypt registered [ 9789.200274] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 9805.818648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9807.252367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9807.325822] alg: No test for adler32 (adler32-zlib) [ 9808.408627] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 9808.640640] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9810.337213] Key type lgssc registered [ 9811.575573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9857.839349] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9869.621486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9869.662353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.258239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9871.365101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9871.534871] Lustre: lustre-MDT0000: new disk, initializing [ 9871.731161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9871.847933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9876.996782] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9888.456458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9888.649885] Lustre: 252959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9888.773634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9888.776741] Lustre: Skipped 1 previous similar message [ 9888.933743] Lustre: lustre-MDT0001: new disk, initializing [ 9889.055345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9889.118652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9889.155134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9894.221796] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9899.668208] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9907.332229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9907.780784] Lustre: lustre-OST0000: new disk, initializing [ 9907.788277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9907.795507] Lustre: 254589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9907.949418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.414333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9912.426862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9912.558118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9914.263934] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9925.518582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9925.767707] Lustre: lustre-OST0001: new disk, initializing [ 9925.778591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9925.789675] Lustre: 255459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9925.952368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9932.732178] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9934.399729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9934.411426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9934.531709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9945.729319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9958.636280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9971.168791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9971.179231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.223853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.785806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.787152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.814184] Lustre: Skipped 2 previous similar messages [ 9977.439914] Lustre: server umount lustre-MDT0000 complete [ 9981.521419] LustreError: 252950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382757 with bad export cookie 1071539481469948851 [ 9981.537429] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.545451] LustreError: 252950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9982.116384] Lustre: server umount lustre-MDT0001 complete [ 9996.728468] Lustre: server umount lustre-OST0000 complete [10011.493371] Lustre: server umount lustre-OST0001 complete [10025.282000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10035.010878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10043.150224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10053.488158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10074.125889] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10084.672548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.693614] Lustre: lustre-MDT0000: reset Object Index mappings [10085.074901] LustreError: 260474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10085.088622] LustreError: 260474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10085.139300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10089.459152] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10090.470889] LustreError: 260475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.565187] LustreError: 260474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10099.104698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10099.131954] Lustre: lustre-MDT0001: reset Object Index mappings [10099.598494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10104.551359] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10107.566182] Lustre: 261618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10114.603866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.634641] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10124.136185] LustreError: 261973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.379351] LustreError: 261975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.413996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.581330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10129.588332] Lustre: Skipped 1 previous similar message [10130.624185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10130.629752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10135.220735] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10142.700117] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10146.692820] Lustre: 263487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10154.265224] Lustre: server umount lustre-MDT0000 complete [10156.002182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10156.014251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.026497] Lustre: Skipped 2 previous similar messages [10156.036622] LustreError: 261731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10157.886047] LustreError: 260456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382934 with bad export cookie 1071539481469955641 [10157.892913] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10157.901418] LustreError: 260456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10158.270286] Lustre: server umount lustre-MDT0001 complete [10172.607473] Lustre: server umount lustre-OST0000 complete [10187.138230] Lustre: server umount lustre-OST0001 complete [10205.785353] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:16:20 (1787382980) [10222.769206] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [10234.664622] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10274.688658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.009245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10275.032722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10275.098232] Lustre: lustre-MDT0000: new disk, initializing [10275.166415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10275.180334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10280.635076] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10291.638365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.830272] Lustre: 267742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10291.884371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10291.893339] Lustre: Skipped 1 previous similar message [10292.013377] Lustre: lustre-MDT0001: new disk, initializing [10292.222287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10292.240877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10297.378680] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10306.824390] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10312.225914] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10320.779790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.186808] Lustre: lustre-OST0000: new disk, initializing [10321.194048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10321.207076] Lustre: 269710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10322.475721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10322.493409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10322.627179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10328.226350] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10337.985235] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10343.409333] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10343.571728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10348.101883] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10348.258206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10353.636292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.647614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10353.657215] Lustre: Skipped 3 previous similar messages [10353.673712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10353.690865] Lustre: Skipped 1 previous similar message [10358.764402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10358.773375] Lustre: Skipped 1 previous similar message [10363.880567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.891981] Lustre: Skipped 1 previous similar message [10364.384130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10364.794116] Lustre: server umount lustre-OST0000 complete [10368.993061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10369.002574] Lustre: Skipped 1 previous similar message [10374.115107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10374.117886] Lustre: Skipped 3 previous similar messages [10383.327141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10383.685369] Lustre: server umount lustre-MDT0000 complete [10384.354131] LustreError: 267754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10384.363301] LustreError: 267754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10387.181204] LustreError: 268538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383163 with bad export cookie 1071539481469957734 [10387.188326] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.193645] LustreError: 268538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10387.511374] Lustre: server umount lustre-MDT0001 complete [10405.574127] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10414.746136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.254519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10415.262441] Lustre: Skipped 2 previous similar messages [10419.792650] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10420.705626] LustreError: 272647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.720473] LustreError: 272647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10428.458462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.599531] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10436.333942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10443.415192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.824908] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10458.062986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10458.190407] Lustre: lustre-OST0001: new disk, initializing [10458.195645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10458.205387] Lustre: 274957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10459.967611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10459.986142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10460.108189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10463.973614] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10473.310378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10476.991472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10482.018689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10482.038281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10490.336561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10490.355669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.374136] Lustre: Skipped 1 previous similar message [10490.378600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.390361] Lustre: Skipped 2 previous similar messages [10494.327450] Lustre: server umount lustre-MDT0000 complete [10496.994729] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10497.048416] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10498.226816] LustreError: 272628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383274 with bad export cookie 1071539481469958616 [10498.232480] LustreError: 272628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10498.236958] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10498.790807] Lustre: server umount lustre-MDT0001 complete [10513.074646] Lustre: server umount lustre-OST0000 complete [10526.902260] Lustre: server umount lustre-OST0001 complete [10547.683677] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10560.098871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.500204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10560.505537] Lustre: Skipped 3 previous similar messages [10565.534510] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10575.247679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.784930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10583.021353] Lustre: 279155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10590.007839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10596.370605] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10604.130317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10605.405618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10605.429857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10610.954979] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10618.911945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10622.338624] Lustre: 281035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10627.664628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10627.724544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10637.084434] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:23:31 (1787383411) [10656.752703] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.770580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10656.780720] Lustre: Skipped 6 previous similar messages [10656.806163] Lustre: Skipped 7 previous similar messages [10659.196259] Lustre: server umount lustre-MDT0000 complete [10661.884352] LustreError: 279067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10661.946261] LustreError: 279067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10663.493892] LustreError: 277995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383439 with bad export cookie 1071539481469965385 [10663.515545] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10663.517577] LustreError: 277995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10669.956942] Lustre: server umount lustre-MDT0001 complete [10680.293245] Lustre: server umount lustre-OST0000 complete [10684.347354] Lustre: server umount lustre-OST0001 complete [10691.347990] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [10704.800663] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10759.262551] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10771.768843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.111531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10772.172939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10772.308566] Lustre: lustre-MDT0000: new disk, initializing [10772.531598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10778.531954] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10792.276712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.446601] Lustre: 285281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10792.837625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10798.025620] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10811.135401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.446278] Lustre: 286917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10812.768740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10818.092642] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10829.629557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10829.879130] Lustre: 287788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10830.001912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10830.019230] Lustre: Skipped 6 previous similar messages [10831.586837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10831.600191] Lustre: Skipped 1 previous similar message [10831.685738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10836.515344] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10846.606764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10850.494946] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10850.498225] Lustre: Skipped 1 previous similar message [10866.479871] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:27:20 (1787383640) [10869.729881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10869.738148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.767272] Lustre: Skipped 2 previous similar messages [10869.785910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10869.802668] Lustre: Skipped 4 previous similar messages [10873.084606] Lustre: server umount lustre-MDT0000 complete [10877.460993] LustreError: 287787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383653 with bad export cookie 1071539481469972805 [10877.465472] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10877.482825] LustreError: 287787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10877.961024] Lustre: server umount lustre-MDT0001 complete [10891.823830] Lustre: server umount lustre-OST0000 complete [10894.303168] Lustre: 250465:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383654/real 1787383654] req@ffff9e204bbe7800 x1874206481845888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10896.328968] Lustre: server umount lustre-OST0001 complete [10905.946659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.750585] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10918.997768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.626770] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10932.126584] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10939.505599] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10946.181111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.770211] LustreError: 292072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.802808] LustreError: 292072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10953.525301] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10962.499188] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10990.191888] loop1: detected capacity change from 0 to 204800000 [10990.495488] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10990.516533] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10990.532581] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10997.148637] loop1: detected capacity change from 0 to 204800000 [11003.070162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.205606] LustreError: 290023:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16705: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11003.223985] LustreError: 290023:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16705 [11003.231759] LustreError: 290023:0:(mgs_handler.c:517:mgs_target_reg()) fs16705-MDT0001: Error checking target index: rc = -13 [11003.250581] LustreError: fs16705-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11003.262081] LustreError: 293756:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11003.275684] LustreError: 293756:0:(tgt_mount.c:2129:server_put_super()) no obd fs16705-MDT0001 [11003.281214] LustreError: 293756:0:(tgt_mount.c:116:server_deregister_mount()) fs16705-MDT0001 not registered [11003.334296] Lustre: server umount fs16705-MDT0001 complete [11003.343864] LustreError: 293756:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11013.217120] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11013.432920] Lustre: Setting parameter fs16705-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16705-MDT0001 [11013.518955] Lustre: srv-fs16705-MDT0001: No data found on store. Initialize space. [11013.526941] Lustre: Skipped 4 previous similar messages [11013.663483] Lustre: fs16705-MDT0001: new disk, initializing [11013.669506] Lustre: Skipped 3 previous similar messages [11013.840446] Lustre: srv-fs16705-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11013.847972] Lustre: 294222:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16705-MDT0001-mdtlov: can't create new llog: rc = -115 [11013.863462] LustreError: 294222:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16705-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11019.263569] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11027.030984] Lustre: server umount fs16705-MDT0001 complete [11031.021189] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11041.621812] loop1: detected capacity change from 0 to 204800000 [11047.798671] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11048.358822] Lustre: 295793:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16705-MDT0001-mdtlov: can't create new llog: rc = -115 [11048.372855] LustreError: 295793:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16705-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11052.775977] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11057.658984] Lustre: server umount fs16705-MDT0001 complete [11067.879160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11067.890822] LustreError: Skipped 1 previous similar message [11067.895780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11067.907083] Lustre: Skipped 4 previous similar messages [11067.914740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11073.746187] Lustre: server umount lustre-MDT0000 complete [11083.603886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11083.785801] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11087.805920] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11089.408332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11095.441397] loop1: detected capacity change from 0 to 204800000 [11100.740444] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.177060] Lustre: 298238:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16705-MDT0001-mdtlov: can't create new llog: rc = -115 [11101.189530] LustreError: 298238:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16705-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11105.793429] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11111.339220] Lustre: server umount fs16705-MDT0001 complete [11138.538320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11143.032525] Lustre: server umount lustre-OST0000 complete [11154.177895] Lustre: server umount lustre-MDT0000 complete [11158.581754] LustreError: 290014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383935 with bad export cookie 1071539481469975486 [11158.586564] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11158.588366] LustreError: 290014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11158.934897] Lustre: server umount lustre-MDT0001 complete [11171.141627] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11173.749683] Key type lgssc unregistered [11174.072816] LNet: 300586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11174.080870] LNetError: 300586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11174.092328] LNet: Removed LNI 192.168.201.115@tcp [11175.323239] Key type .llcrypt unregistered [11175.326390] Key type ._llcrypt unregistered [11194.419179] Key type ._llcrypt registered [11194.421768] Key type .llcrypt registered [11194.522877] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11195.397662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11195.502557] alg: No test for adler32 (adler32-zlib) [11196.510644] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11196.762834] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11198.535621] Key type lgssc registered [11199.931671] Lustre: Echo OBD driver; http://www.lustre.org/ [11208.832235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11208.865207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.493234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.009228] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11219.950416] LustreError: 301860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.984623] LustreError: 301860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11224.230092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.878310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11229.071902] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11236.590805] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11243.275279] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11250.276820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.494899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.633564] LustreError: 303688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11256.431401] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11259.885323] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.023650] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11265.001158] LustreError: 303955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.123352] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.132747] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11280.355894] LustreError: 303955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.401000] LustreError: 303955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11285.952635] loop1: detected capacity change from 0 to 204800000 [11286.197590] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11286.206933] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11286.223247] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11291.879424] loop1: detected capacity change from 0 to 204800000 [11297.454823] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.495279] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11297.961238] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11298.058811] Lustre: fs19631-OST0000: new disk, initializing [11298.063981] Lustre: srv-fs19631-OST0000: No data found on store. Initialize space. [11298.069704] Lustre: 305597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11298.138704] Lustre: fs19631-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11304.265310] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11313.140351] Lustre: server umount fs19631-OST0000 complete [11325.941363] loop1: detected capacity change from 0 to 204800000 [11331.570401] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.594609] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11332.424077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.652175] Lustre: fs19631-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.679910] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11338.554179] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11347.919102] loop2: detected capacity change from 0 to 204800000 [11348.206270] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11348.229298] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11348.239688] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11352.050299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11353.218108] loop2: detected capacity change from 0 to 204800000 [11358.323953] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.417033] LustreError: fs19631-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11358.425538] LustreError: fs19631-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11358.431721] LustreError: 308639:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11358.436973] LustreError: 308639:0:(tgt_mount.c:2129:server_put_super()) no obd fs19631-OST0001 [11358.441935] LustreError: 308639:0:(tgt_mount.c:116:server_deregister_mount()) fs19631-OST0001 not registered [11358.484729] Lustre: server umount fs19631-OST0001 complete [11358.487471] LustreError: 308639:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11367.712815] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.856371] Lustre: fs19631-OST0001: new disk, initializing [11367.864550] Lustre: srv-fs19631-OST0001: No data found on store. Initialize space. [11367.871487] Lustre: 309190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11367.974540] Lustre: fs19631-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11374.250935] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11382.615676] Lustre: server umount fs19631-OST0000 complete [11391.095766] Lustre: server umount fs19631-OST0001 complete [11398.130460] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.151330] LustreError: 303690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11404.149658] loop1: detected capacity change from 0 to 204800000 [11410.185557] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11410.401064] Lustre: fs19631-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11417.709502] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11425.038544] loop2: detected capacity change from 0 to 204800000 [11431.471504] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11439.173544] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11451.359825] Lustre: server umount fs19631-OST0000 complete [11461.416566] Lustre: server umount fs19631-OST0001 complete [11474.920971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11474.926074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.942652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11474.949077] Lustre: Skipped 1 previous similar message [11480.055753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11480.078905] Lustre: Skipped 1 previous similar message [11480.920593] Lustre: server umount lustre-OST0000 complete [11485.177196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.190054] Lustre: Skipped 1 previous similar message [11485.203958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.274287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.292538] Lustre: Skipped 2 previous similar messages [11491.496438] Lustre: server umount lustre-MDT0000 complete [11495.596961] LustreError: 301839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384272 with bad export cookie 8123701190760842775 [11495.599659] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11495.612596] LustreError: 301839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11496.186498] Lustre: server umount lustre-MDT0001 complete [11508.581476] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11511.846295] Key type lgssc unregistered [11512.175269] LNet: 315118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11512.183571] LNetError: 315118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11512.195161] LNet: Removed LNI 192.168.201.115@tcp [11513.170220] Key type .llcrypt unregistered [11513.172222] Key type ._llcrypt unregistered [11547.485148] Key type ._llcrypt registered [11547.486607] Key type .llcrypt registered [11547.656709] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [11566.597272] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11568.101094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11568.274405] alg: No test for adler32 (adler32-zlib) [11569.298560] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11569.492864] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11571.143267] Key type lgssc registered [11572.295866] Lustre: Echo OBD driver; http://www.lustre.org/ [11609.731351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11609.755894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.135852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11611.162480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11611.217057] Lustre: lustre-MDT0000: new disk, initializing [11611.280988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11611.294421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11616.475987] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11627.957705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.128798] Lustre: 318979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11628.179838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11628.182921] Lustre: Skipped 1 previous similar message [11628.247365] Lustre: lustre-MDT0001: new disk, initializing [11628.358241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11628.392074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11628.407228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11633.217446] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11643.192106] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11649.631816] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11657.565128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.845448] Lustre: lustre-OST0000: new disk, initializing [11657.856309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11657.866854] Lustre: 320951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11657.985633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11665.064131] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11666.496646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11666.538293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11666.654670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11675.689544] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11678.222586] Lustre: Mounted lustre-client - version 2.17.57_45_g4630995 [11679.349511] Lustre: Unmounted lustre-client [11692.513319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11692.528321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.546857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11693.541636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11693.550122] Lustre: Skipped 2 previous similar messages [11695.845539] Lustre: server umount lustre-OST0000 complete [11700.192520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11700.197681] LustreError: Skipped 1 previous similar message [11700.205055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.221795] Lustre: Skipped 1 previous similar message [11700.231677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.776960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.778695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.904263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11708.912510] Lustre: Skipped 2 previous similar messages [11714.017093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11714.321262] Lustre: server umount lustre-MDT0000 complete [11717.864265] LustreError: 318974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384494 with bad export cookie 825387436431530791 [11717.873100] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.392460] Lustre: server umount lustre-MDT0001 complete [11734.881895] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11738.274642] Key type lgssc unregistered [11738.792268] LNet: 323139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11738.803213] LNetError: 323139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11810.537721] LNet: Removed LNI 192.168.201.115@tcp [11811.661217] Key type .llcrypt unregistered [11811.663295] Key type ._llcrypt unregistered [11830.481332] Key type ._llcrypt registered [11830.482938] Key type .llcrypt registered [11830.640491] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [11847.610937] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11848.669174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11848.865330] alg: No test for adler32 (adler32-zlib) [11850.016204] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11850.224488] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11851.880945] Key type lgssc registered [11853.161087] Lustre: Echo OBD driver; http://www.lustre.org/ [11885.385580] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:44:19 (1787384659) [11892.529803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11892.561090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.901430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11893.930085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11894.023218] Lustre: lustre-MDT0000: new disk, initializing [11894.154436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11894.220698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11899.083467] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11910.672703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.825454] Lustre: 326363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11910.910624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11910.915254] Lustre: Skipped 1 previous similar message [11910.983616] Lustre: lustre-MDT0001: new disk, initializing [11911.129495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11911.173101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11911.187194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11916.483510] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11926.437931] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11933.052642] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11939.603600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.844820] Lustre: lustre-OST0000: new disk, initializing [11939.848417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11939.856535] Lustre: 328331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11939.924628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11947.178827] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11948.181046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11948.189585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11948.377513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11958.085318] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11974.112581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11974.118309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11974.123771] LustreError: Skipped 1 previous similar message [11974.126659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11974.163659] Lustre: Skipped 1 previous similar message [11979.253658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.265405] Lustre: Skipped 2 previous similar messages [11980.069218] Lustre: server umount lustre-OST0000 complete [11987.937231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11987.944091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11987.956344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11989.498963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.071739] Lustre: server umount lustre-MDT0000 complete [11995.516163] LustreError: 327158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384771 with bad export cookie 201932865314845413 [11995.518376] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11995.535276] LustreError: 327158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11995.845576] Lustre: server umount lustre-MDT0001 complete [12008.862707] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12012.033457] Key type lgssc unregistered [12012.399835] LNet: 330243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12012.403432] LNetError: 330243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12012.417154] LNet: Removed LNI 192.168.201.115@tcp [12013.302372] Key type .llcrypt unregistered [12013.305687] Key type ._llcrypt unregistered [12036.397655] Key type ._llcrypt registered [12036.401249] Key type .llcrypt registered [12036.527464] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12037.782988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12037.814284] alg: No test for adler32 (adler32-zlib) [12038.852734] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12039.107571] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12040.799439] Key type lgssc registered [12042.334679] Lustre: Echo OBD driver; http://www.lustre.org/ [12050.996104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12051.025365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12052.516318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12057.263970] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12061.666593] LustreError: 331517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.683610] LustreError: 331517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12066.784199] LustreError: 331516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.533053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.879424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12072.700552] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12080.608851] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12086.507099] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12094.209787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.431179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.808341] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12109.407675] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12123.951253] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12125.862310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12133.041550] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [12133.348546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12133.375911] Lustre: Skipped 1 previous similar message [12136.170167] Lustre: server umount lustre-MDT0000 complete [12138.122658] LustreError: 333516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.947439] LustreError: 331499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384916 with bad export cookie 963587651461636926 [12139.958132] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12143.257173] LustreError: 333516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12143.287832] LustreError: 333516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12143.306348] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12143.310975] Lustre: Skipped 3 previous similar messages [12143.585727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.599691] Lustre: Skipped 1 previous similar message [12144.610640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12144.629087] Lustre: Skipped 1 previous similar message [12148.367488] LustreError: 333516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12148.368839] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12148.390247] LustreError: 333516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12153.499861] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12153.508783] Lustre: Skipped 1 previous similar message [12154.335308] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12154.579688] Lustre: server umount lustre-MDT0001 complete [12158.390730] Lustre: server umount lustre-OST0000 complete [12167.390076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.703491] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12175.544861] Lustre: 335699:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.115@tcp [12177.853556] Lustre: server umount MGS complete [12188.702340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12189.263296] LustreError: 336273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.288383] LustreError: 336273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12189.331207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12193.697688] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12203.327595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.601466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.008628] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12216.960712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12217.312685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12224.657588] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12232.092377] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12233.801802] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12239.961030] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12242.024638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12248.057521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12248.072142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.102905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.939820] Lustre: server umount lustre-OST0000 complete [12254.177450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.198142] Lustre: Skipped 1 previous similar message [12259.646861] Lustre: server umount lustre-MDT0000 complete [12262.824128] LustreError: 336253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385039 with bad export cookie 963587651461637766 [12262.827877] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12262.837241] LustreError: 336253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12273.377546] Lustre: server umount lustre-MDT0001 complete [12285.799168] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12288.513492] Key type lgssc unregistered [12288.765781] LNet: 339714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12288.779513] LNetError: 339714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12288.798608] LNet: Removed LNI 192.168.201.115@tcp [12289.575776] Key type .llcrypt unregistered [12289.577632] Key type ._llcrypt unregistered [12311.690947] Key type ._llcrypt registered [12311.699891] Key type .llcrypt registered [12311.826697] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [12330.737525] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12332.210851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12332.244593] alg: No test for adler32 (adler32-zlib) [12333.358526] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12333.647143] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12335.383612] Key type lgssc registered [12336.283695] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.153320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.192056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.417500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12376.440633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12376.517567] Lustre: lustre-MDT0000: new disk, initializing [12376.591412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12376.606488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12381.308847] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12391.462957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12391.595265] Lustre: 342881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12391.645491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12391.649897] Lustre: Skipped 1 previous similar message [12391.697861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12391.751388] Lustre: lustre-MDT0001: new disk, initializing [12391.823375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12391.862851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12391.875137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12396.669596] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12407.109207] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12414.186332] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12420.210825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12420.576076] Lustre: lustre-OST0000: new disk, initializing [12420.587671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12420.598670] Lustre: 344846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12420.692113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12427.867326] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12427.873387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12427.886510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12428.011221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12438.445180] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12442.960868] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12443.195552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12447.729161] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12447.944647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12453.859422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12453.865232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.876185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.882267] Lustre: Skipped 1 previous similar message [12458.978900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12458.984215] Lustre: Skipped 1 previous similar message [12464.095532] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12464.100245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12464.133234] Lustre: Skipped 1 previous similar message [12464.292663] Lustre: server umount lustre-OST0000 complete [12468.706899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12468.713121] LustreError: Skipped 1 previous similar message [12468.725918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.743409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12468.748653] Lustre: Skipped 1 previous similar message [12474.246571] Lustre: server umount lustre-MDT0000 complete [12477.820658] LustreError: 342873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385254 with bad export cookie 10709924530266838490 [12477.821892] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.830408] LustreError: 342873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12478.171542] Lustre: server umount lustre-MDT0001 complete [12487.751177] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:54:21 (1787385261) [12503.818177] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [12514.796379] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12554.452597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.659374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12554.688241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12554.763784] Lustre: lustre-MDT0000: new disk, initializing [12554.909100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12554.941299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12559.197537] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12569.140684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12569.267481] Lustre: 349608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12569.319260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12569.323665] Lustre: Skipped 1 previous similar message [12569.383406] Lustre: lustre-MDT0001: new disk, initializing [12569.508744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12569.591358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12569.597940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12573.822486] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12582.745979] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12588.323738] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12596.381469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12596.744828] Lustre: lustre-OST0000: new disk, initializing [12596.752815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12596.762792] Lustre: 351574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12596.904943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12603.454609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12603.459854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12603.569278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12603.884538] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12613.940183] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12655.075099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12655.075122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.084983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12655.084991] Lustre: Skipped 2 previous similar messages [12655.090270] LustreError: Skipped 1 previous similar message [12655.134122] Lustre: Skipped 2 previous similar messages [12658.055919] Lustre: server umount lustre-OST0000 complete [12665.321056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.341301] Lustre: Skipped 1 previous similar message [12668.595059] Lustre: server umount lustre-MDT0000 complete [12670.435100] LustreError: 349613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.464522] LustreError: 349613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12673.103663] LustreError: 349600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385449 with bad export cookie 10709924530266839372 [12673.116742] LustreError: 349600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12673.118244] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12673.467242] Lustre: server umount lustre-MDT0001 complete [12684.409504] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12687.299390] Key type lgssc unregistered [12687.668133] LNet: 353750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12687.679755] LNetError: 353750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12687.693653] LNet: Removed LNI 192.168.201.115@tcp [12688.575455] Key type .llcrypt unregistered [12688.583804] Key type ._llcrypt unregistered [12786.480347] Key type ._llcrypt registered [12786.482348] Key type .llcrypt registered [12786.561118] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12786.800740] Key type .llcrypt unregistered [12786.806323] Key type ._llcrypt unregistered [12868.269662] Key type ._llcrypt registered [12868.273794] Key type .llcrypt registered [12868.395908] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12868.718428] Key type .llcrypt unregistered [12868.723481] Key type ._llcrypt unregistered [12903.100757] Key type ._llcrypt registered [12903.102186] Key type .llcrypt registered [12903.252387] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12903.690366] Key type .llcrypt unregistered [12903.692071] Key type ._llcrypt unregistered [12935.898480] Key type ._llcrypt registered [12935.901580] Key type .llcrypt registered [12936.032798] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12936.321380] Key type .llcrypt unregistered [12936.324648] Key type ._llcrypt unregistered [13021.235171] Key type ._llcrypt registered [13021.237280] Key type .llcrypt registered [13021.373132] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13022.014392] Key type .llcrypt unregistered [13022.016302] Key type ._llcrypt unregistered [13059.180190] Key type ._llcrypt registered [13059.182528] Key type .llcrypt registered [13059.279141] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13059.593552] Key type .llcrypt unregistered [13059.597505] Key type ._llcrypt unregistered [13095.092882] Key type ._llcrypt registered [13095.104160] Key type .llcrypt registered [13095.293863] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13095.953380] Key type .llcrypt unregistered [13095.958432] Key type ._llcrypt unregistered [13127.545567] Key type ._llcrypt registered [13127.553421] Key type .llcrypt registered [13127.709679] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13127.993330] Key type .llcrypt unregistered [13127.995417] Key type ._llcrypt unregistered [13159.547272] Key type ._llcrypt registered [13159.549384] Key type .llcrypt registered [13159.643973] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13160.087472] Key type .llcrypt unregistered [13160.089133] Key type ._llcrypt unregistered [13196.095110] Key type ._llcrypt registered [13196.101955] Key type .llcrypt registered [13196.349299] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13196.816330] Key type .llcrypt unregistered [13196.818516] Key type ._llcrypt unregistered [13265.489414] Key type ._llcrypt registered [13265.492226] Key type .llcrypt registered [13265.622516] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13266.064435] Key type .llcrypt unregistered [13266.066441] Key type ._llcrypt unregistered [13299.081110] Key type ._llcrypt registered [13299.083285] Key type .llcrypt registered [13299.135645] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13299.356530] Key type .llcrypt unregistered [13299.359334] Key type ._llcrypt unregistered [13343.809860] Key type ._llcrypt registered [13343.812944] Key type .llcrypt registered [13343.946792] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13344.306545] Key type .llcrypt unregistered [13344.308656] Key type ._llcrypt unregistered [13409.951931] Key type ._llcrypt registered [13409.954309] Key type .llcrypt registered [13410.118146] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [13427.592353] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [13429.372926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13429.526613] alg: No test for adler32 (adler32-zlib) [13430.864312] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [13431.306261] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [13433.087685] Key type lgssc registered [13435.143670] Lustre: Echo OBD driver; http://www.lustre.org/ [13480.124600] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13480.160682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13481.502351] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13481.571923] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13481.684898] Lustre: tst250fs-MDT0000: new disk, initializing [13481.756972] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13481.774212] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13486.219131] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13497.545861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13497.845995] Lustre: 374166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13497.893144] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13497.896663] Lustre: Skipped 1 previous similar message [13497.960971] Lustre: tst250fs-MDT0001: new disk, initializing [13498.087775] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13498.137519] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13498.152387] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13503.335721] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13508.522328] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13516.117291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13516.491530] Lustre: tst250fs-OST0000: new disk, initializing [13516.494954] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13516.500972] Lustre: 375803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13516.596412] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13522.034539] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13522.047661] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13522.181049] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13524.678909] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13536.103368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13536.327613] Lustre: tst250fs-OST0001: new disk, initializing [13536.341950] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13536.360438] Lustre: 376674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13536.509476] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13541.502066] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13541.514588] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13541.664336] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13543.668370] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13549.331368] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13551.736131] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13551.739339] Lustre: Skipped 2 previous similar messages [13572.593305] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13572.607445] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13572.619481] Lustre: Skipped 1 previous similar message [13572.635820] Lustre: Skipped 2 previous similar messages [13577.708916] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13577.720687] Lustre: Skipped 4 previous similar messages [13582.835926] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13582.845503] Lustre: Skipped 3 previous similar messages [13586.399759] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13586.714834] Lustre: server umount tst250fs-MDT0000 complete [13590.817716] LustreError: 374161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386367 with bad export cookie 1095400333007213610 [13590.820898] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13590.829265] LustreError: 374161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13591.088475] Lustre: server umount tst250fs-MDT0001 complete [13604.940053] Lustre: server umount tst250fs-OST0000 complete [13608.930950] Lustre: 372390:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386369/real 1787386369] req@ffff9e215178a680 x1874210279462528/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787386385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13608.971133] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13609.001669] Lustre: Skipped 2 previous similar messages [13609.714087] Lustre: server umount tst250fs-OST0001 complete [13629.874846] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [13640.921619] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [13678.706435] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:14:12 (1787386452) [13680.411757] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13704.751447] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [13723.722668] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [13765.989266] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13380 sec ========= 04:15:39 (1787386539) [13768.664675] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:15:42 (1787386542) === [13773.055961] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:15:46 (1787386546) === [13819.173930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13822.304852] Key type lgssc unregistered [13822.553385] LNet: 385829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13822.562876] LNetError: 385829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13823.589574] LNet: Removed LNI 192.168.201.115@tcp [13824.691293] Key type .llcrypt unregistered [13824.696048] Key type ._llcrypt unregistered