[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490789389 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003232] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008789] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011177] LSM: Security Framework initializing [ 0.012080] Yama: becoming mindful. [ 0.013058] SELinux: Initializing. [ 0.014090] *** VALIDATE selinux *** [ 0.022723] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027471] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028199] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030148] *** VALIDATE tmpfs *** [ 0.032376] *** VALIDATE proc *** [ 0.033362] *** VALIDATE cgroup *** [ 0.034020] *** VALIDATE cgroup2 *** [ 0.036003] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037196] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039037] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.043124] debug: unmapping init [mem 0xffffffffa2e59000-0xffffffffa2e60fff] [ 0.045218] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053028] ... event mask: 000000070000000f [ 0.055279] rcu: Hierarchical SRCU implementation. [ 0.057652] smp: Bringing up secondary CPUs ... [ 0.058600] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.062448] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231028] node 0 deferred pages initialised in 163ms [ 0.235159] devtmpfs: initialized [ 0.236274] x86/mm: Memory block size: 128MB [ 0.239000] gcov: version magic: 0x41383552 [ 0.242435] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249462] pinctrl core: initialized pinctrl subsystem [ 0.252199] [ 0.252685] ************************************************************* [ 0.255019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258018] ** ** [ 0.260017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263017] ** ** [ 0.265015] ** This means that this kernel is built to expose internal ** [ 0.268015] ** IOMMU data structures, which may compromise security on ** [ 0.270016] ** your system. ** [ 0.273020] ** ** [ 0.276017] ** If you see this message and you are not debugging the ** [ 0.278015] ** kernel, report this immediately to your vendor! ** [ 0.280018] ** ** [ 0.283016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285020] ************************************************************* [ 0.289218] NET: Registered protocol family 16 [ 0.291522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.298083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302134] cpuidle: using governor menu [ 0.305649] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.308640] PCI: Using configuration type 1 for base access [ 0.310217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326106] cryptd: max_cpu_qlen set to 1000 [ 0.329399] ACPI: Added _OSI(Module Device) [ 0.331021] ACPI: Added _OSI(Processor Device) [ 0.333018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.334017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.340604] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.349605] ACPI: Interpreter enabled [ 0.351088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352016] ACPI: Using IOAPIC for interrupt routing [ 0.354141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358509] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370542] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.375025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.385030] acpiphp: Slot [2] registered [ 0.387398] acpiphp: Slot [5] registered [ 0.389207] acpiphp: Slot [6] registered [ 0.391200] acpiphp: Slot [7] registered [ 0.392166] acpiphp: Slot [8] registered [ 0.396236] acpiphp: Slot [9] registered [ 0.398212] acpiphp: Slot [10] registered [ 0.399185] acpiphp: Slot [3] registered [ 0.401140] acpiphp: Slot [4] registered [ 0.403140] acpiphp: Slot [11] registered [ 0.405156] acpiphp: Slot [12] registered [ 0.406126] acpiphp: Slot [13] registered [ 0.408127] acpiphp: Slot [14] registered [ 0.409135] acpiphp: Slot [15] registered [ 0.410116] acpiphp: Slot [16] registered [ 0.412133] acpiphp: Slot [17] registered [ 0.413163] acpiphp: Slot [18] registered [ 0.415147] acpiphp: Slot [19] registered [ 0.417149] acpiphp: Slot [20] registered [ 0.419169] acpiphp: Slot [21] registered [ 0.421141] acpiphp: Slot [22] registered [ 0.423175] acpiphp: Slot [23] registered [ 0.424126] acpiphp: Slot [24] registered [ 0.426137] acpiphp: Slot [25] registered [ 0.428135] acpiphp: Slot [26] registered [ 0.429130] acpiphp: Slot [27] registered [ 0.431134] acpiphp: Slot [28] registered [ 0.433139] acpiphp: Slot [29] registered [ 0.434159] acpiphp: Slot [30] registered [ 0.436140] acpiphp: Slot [31] registered [ 0.438084] PCI host bridge to bus 0000:00 [ 0.439026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.447051] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.450036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.454042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.456268] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460110] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.463594] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.473761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.479528] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.483037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.485023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.487026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.491692] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.494868] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.497052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.500063] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.505023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.521023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.528022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535287] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.543018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.549017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.564020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.574824] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.582023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.607018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.617300] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634753] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.651018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.686022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.711225] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.727317] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.741017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.772016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.785406] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.793016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.805020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.839017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.853278] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.868022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.888021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.916019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.950914] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.953958] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.957819] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.962052] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.966289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.974085] iommu: Default domain type: Passthrough [ 0.984573] SCSI subsystem initialized [ 0.988611] ACPI: bus type USB registered [ 0.992413] usbcore: registered new interface driver usbfs [ 0.997175] usbcore: registered new interface driver hub [ 1.003248] usbcore: registered new device driver usb [ 1.005022] pps_core: LinuxPPS API ver. 1 registered [ 1.007013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.011218] PTP clock support registered [ 1.014095] EDAC MC: Ver: 3.0.0 [ 1.017410] PCI: Using ACPI for IRQ routing [ 1.021055] NetLabel: Initializing [ 1.024016] NetLabel: domain hash size = 128 [ 1.026087] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.030098] NetLabel: unlabeled traffic allowed by default [ 1.033604] vgaarb: loaded [ 1.035369] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.037014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.046595] clocksource: Switched to clocksource kvm-clock [ 1.303619] VFS: Disk quotas dquot_6.6.0 [ 1.313948] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.322559] *** VALIDATE ramfs *** [ 1.325928] *** VALIDATE hugetlbfs *** [ 1.329698] pnp: PnP ACPI init [ 1.339522] pnp: PnP ACPI: found 6 devices [ 1.379446] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.388922] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.397677] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.404373] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.409410] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.417483] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.427892] NET: Registered protocol family 2 [ 1.431182] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.442047] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.447832] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.456476] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.461377] TCP: Hash tables configured (established 65536 bind 65536) [ 1.466703] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.470572] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.473965] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.477550] NET: Registered protocol family 1 [ 1.485086] RPC: Registered named UNIX socket transport module. [ 1.491422] RPC: Registered udp transport module. [ 1.496397] RPC: Registered tcp transport module. [ 1.508969] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.517681] NET: Registered protocol family 44 [ 1.519343] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.522603] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.525861] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.529381] PCI: CLS 0 bytes, default 64 [ 1.531649] Unpacking initramfs... [ 4.824943] debug: unmapping init [mem 0xffff8b667cc54000-0xffff8b667ffbffff] [ 4.845948] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.848772] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.854032] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.180833] Initialise system trusted keyrings [ 6.196615] Key type blacklist registered [ 6.205510] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.224223] zbud: loaded [ 6.231326] *** VALIDATE nfs *** [ 6.236100] *** VALIDATE nfs4 *** [ 6.242055] pstore: using deflate compression [ 6.249726] Platform Keyring initialized [ 6.510991] NET: Registered protocol family 38 [ 6.513672] Key type asymmetric registered [ 6.520794] Asymmetric key parser 'x509' registered [ 6.522659] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.531356] io scheduler mq-deadline registered [ 6.533725] io scheduler kyber registered [ 6.537672] io scheduler bfq registered [ 6.540984] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.551590] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.570393] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.578488] ACPI: Power Button [PWRF] [ 6.587437] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.609680] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.634720] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.661175] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.690133] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.727085] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.768967] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.777544] Non-volatile memory driver v1.3 [ 6.780581] Linux agpgart interface v0.103 [ 6.840359] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 6.844468] vda: detected capacity change from 0 to 74698752 [ 6.865500] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.874082] vdb: detected capacity change from 0 to 1073741824 [ 6.910589] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.917374] vdc: detected capacity change from 0 to 2621440000 [ 6.959860] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.976317] vdd: detected capacity change from 0 to 2621440000 [ 7.017692] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.022380] vde: detected capacity change from 0 to 4294967296 [ 7.085897] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.095604] vdf: detected capacity change from 0 to 4294967296 [ 7.108137] libphy: Fixed MDIO Bus: probed [ 7.122547] usbcore: registered new interface driver usbserial_generic [ 7.125476] usbserial: USB Serial support registered for generic [ 7.127939] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.135330] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.140482] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.152279] mousedev: PS/2 mouse device common for all mice [ 7.155387] rtc_cmos 00:05: RTC can wake from S4 [ 7.163833] rtc_cmos 00:05: registered as rtc0 [ 7.166423] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.173093] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.174818] intel_pstate: CPU model not supported [ 7.196515] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.207847] hid: raw HID events driver (C) Jiri Kosina [ 7.208059] usbcore: registered new interface driver usbhid [ 7.208063] usbhid: USB HID core driver [ 7.215394] drop_monitor: Initializing network drop monitor service [ 7.227983] Initializing XFRM netlink socket [ 7.237409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.241628] NET: Registered protocol family 10 [ 7.261473] Segment Routing with IPv6 [ 7.264761] NET: Registered protocol family 17 [ 7.268470] mpls_gso: MPLS GSO support [ 7.279716] RAS: Correctable Errors collector initialized. [ 7.287766] AVX version of gcm_enc/dec engaged. [ 7.291380] AES CTR mode by8 optimization enabled [ 7.431467] sched_clock: Marking stable (7431439320, 0)->(8430126621, -998687301) [ 7.440727] registered taskstats version 1 [ 7.448502] Loading compiled-in X.509 certificates [ 7.451469] zswap: loaded using pool lzo/zbud [ 7.498882] Key type big_key registered [ 7.531919] Key type encrypted registered [ 7.533609] ima: No TPM chip found, activating TPM-bypass! [ 7.537331] ima: Allocated hash algorithm: sha1 [ 7.542512] ima: No architecture policies found [ 7.546502] evm: Initialising EVM extended attributes: [ 7.549342] evm: security.selinux [ 7.551632] evm: security.ima [ 7.553348] evm: security.capability [ 7.555330] evm: HMAC attrs: 0x1 [ 7.559320] rtc_cmos 00:05: setting system clock to 2026-09-03 13:18:51 UTC (1788441531) [ 7.569602] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 7.574609] debug: unmapping init [mem 0xffffffffa2b82000-0xffffffffa2e58fff] [ 7.591343] Write protecting the kernel read-only data: 28672k [ 7.604732] debug: unmapping init [mem 0xffffffffa1203000-0xffffffffa13fffff] [ 7.610701] debug: unmapping init [mem 0xffffffffa1b14000-0xffffffffa1bfffff] [ 7.742491] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.765612] systemd[1]: Detected virtualization kvm. [ 7.772442] systemd[1]: Detected architecture x86-64. [ 7.784314] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.827839] systemd[1]: No hostname configured. [ 7.830760] systemd[1]: Set hostname to . [ 7.833878] random: systemd: uninitialized urandom read (16 bytes read) [ 7.837354] systemd[1]: Initializing machine ID from random generator. [ 7.960362] random: ln: uninitialized urandom read (6 bytes read) [ 8.151790] random: systemd: uninitialized urandom read (16 bytes read) [ 8.155607] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 8.164821] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.187751] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.879575] device-mapper: uevent: version 1.0.3 [ 9.891876] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 11.560380] random: fast init done [ 11.815926] virtio_net virtio0 ens2: renamed from eth0 [ 12.093509] scsi host0: ata_piix [ 12.101040] scsi host1: ata_piix [ 12.107488] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.109898] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.079253] random: crng init done [ 17.083028] random: 7 urandom warning(s) missed due to ratelimiting [ 17.840174] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.225682] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.627606] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.209854] SELinux: Disabled at runtime. [ 23.334786] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.364670] systemd[1]: Detected virtualization kvm. [ 23.370392] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.493788] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.499821] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.517075] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.530523] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.545295] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.565965] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.589845] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [[ 24.839377] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 25.758384] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.707549] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.799105] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.226882] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.333827] EDAC sbridge: Ver: 1.1.2 [ 31.182464] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only[ 31.967187] NFS: Registering the id_resolver key type root support (7s / no limit)[ 31.974208] Key type id_resolver registered [ 31.982149] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg659-server login: [ 67.969110] hrtimer: interrupt took 3386898 ns [ 84.042432] libcfs: loading out-of-tree module taints kernel. [ 84.171398] Key type ._llcrypt registered [ 84.174191] Key type .llcrypt registered [ 84.327816] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 103.383552] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 105.293070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.321225] alg: No test for adler32 (adler32-zlib) [ 106.814316] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 107.869122] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 109.735627] Key type lgssc registered [ 111.447119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.087541] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.099444] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 185.323897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.359222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.698090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.721277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.827496] Lustre: lustre-MDT0000: new disk, initializing [ 186.891812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.908899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.365316] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.217332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.297937] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.336510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.339969] Lustre: Skipped 1 previous similar message [ 204.416342] Lustre: lustre-MDT0001: new disk, initializing [ 204.494091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.526517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.536380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.102071] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.047854] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.527119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.699363] Lustre: lustre-OST0000: new disk, initializing [ 222.702993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.706114] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.737402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.799644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.822271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.930394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.916831] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.727244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.855407] Lustre: lustre-OST0001: new disk, initializing [ 244.859525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.864815] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.924315] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.677967] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.571633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.591757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.707140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.649235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.547232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.257245] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.945783] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing yml_node [ 286.181248] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 288.462470] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 290.774272] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 292.201171] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:23:34 EDT 2026 [ 305.540940] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 306.671951] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 322.527734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 322.533847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.550376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.600129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.601447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.999243] Lustre: server umount lustre-MDT0000 complete [ 331.621279] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441855 with bad export cookie 17611793797905461056 [ 331.625390] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.630165] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 335.840175] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.842011] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.847106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.847123] Lustre: Skipped 3 previous similar messages [ 335.867483] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 335.910546] Lustre: Skipped 3 previous similar messages [ 336.864579] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.890994] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 338.270048] Lustre: server umount lustre-MDT0001 complete [ 342.002472] Lustre: server umount lustre-OST0000 complete [ 345.786341] Lustre: server umount lustre-OST0001 complete [ 352.363395] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 361.767853] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 395.101369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.426281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.470418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.575242] Lustre: lustre-MDT0000: new disk, initializing [ 395.707972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.736343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.225661] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 409.742024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.885749] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 409.949714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 409.957431] Lustre: Skipped 1 previous similar message [ 410.060250] Lustre: lustre-MDT0001: new disk, initializing [ 410.134558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.169498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.178270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.159518] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 423.172828] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.615584] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 435.062702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 435.407196] Lustre: lustre-OST0000: new disk, initializing [ 435.410735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 435.419768] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 435.513479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.564198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 436.580619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 436.724717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.669140] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 450.788318] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 455.530872] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.771343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.568820] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.775160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 467.428457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.433501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.438386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.446416] Lustre: Skipped 1 previous similar message [ 467.458980] Lustre: Skipped 4 previous similar messages [ 468.791921] Lustre: server umount lustre-OST0000 complete [ 472.549390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.552948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.558707] Lustre: Skipped 1 previous similar message [ 472.568263] Lustre: Skipped 1 previous similar message [ 482.785636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.794101] Lustre: Skipped 3 previous similar messages [ 486.376526] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.652421] Lustre: server umount lustre-MDT0000 complete [ 487.908510] LustreError: 15958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.921940] LustreError: 15958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 489.792644] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442013 with bad export cookie 17611793797905462596 [ 489.807682] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.124368] Lustre: server umount lustre-MDT0001 complete [ 494.932526] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:26:57 (1788442017) [ 511.663828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.988230] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.044841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.730384] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 517.091269] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.105680] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 522.932204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.902873] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 534.009856] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.013305] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 542.220186] Lustre: Failing over lustre-MDT0000 [ 542.550391] Lustre: server umount lustre-MDT0000 complete [ 543.714858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 543.728709] LustreError: Skipped 1 previous similar message [ 543.732106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.745618] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.760357] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 561.719094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.851783] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.422157] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 567.270626] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 567.278352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 567.306883] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 575.515355] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 581.610165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.613567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 581.624082] Lustre: Skipped 2 previous similar messages [ 581.647563] Lustre: Skipped 1 previous similar message [ 593.375155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.529722] Lustre: server umount lustre-MDT0000 complete [ 593.891748] LustreError: 20977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.913922] LustreError: 20977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 596.854961] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442120 with bad export cookie 17611793797905464010 [ 596.868667] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.873743] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 597.548905] Lustre: server umount lustre-MDT0001 complete [ 605.436264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.833182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.842525] Lustre: Skipped 2 previous similar messages [ 610.051162] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 617.906535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.454886] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 629.633425] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 635.302110] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 637.306869] Lustre: Failing over lustre-MDT0000 [ 637.507220] Lustre: server umount lustre-MDT0000 complete [ 638.949227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.962334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.978284] LustreError: 23253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.999554] LustreError: 23253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 654.876266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.051123] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.308531] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.540668] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 660.458386] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 660.460898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 660.470932] Lustre: Skipped 1 previous similar message [ 660.491784] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 666.614939] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 670.691844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.694451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 670.695825] Lustre: Skipped 4 previous similar messages [ 676.556565] Lustre: server umount lustre-MDT0000 complete [ 679.971534] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442203 with bad export cookie 17611793797905464948 [ 679.986301] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.988057] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 680.277662] Lustre: server umount lustre-MDT0001 complete [ 687.872802] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:30:10 (1788442210) [ 700.984728] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 709.402777] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 742.947606] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 751.966908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.146952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.179922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.254166] Lustre: lustre-MDT0000: new disk, initializing [ 752.318818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.325300] Lustre: Skipped 2 previous similar messages [ 752.344570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 756.162437] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 765.465797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.584478] Lustre: 29629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 765.781491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 770.053932] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 778.408153] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 783.321561] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 789.198247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 789.491547] Lustre: 31595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 791.167146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 791.209098] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 791.316081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 791.330466] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 796.050196] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 805.454972] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 810.652162] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 832.480504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 832.488969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.495804] Lustre: Skipped 4 previous similar messages [ 832.498456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 832.501789] Lustre: Skipped 3 previous similar messages [ 833.880801] Lustre: server umount lustre-OST0000 complete [ 844.100885] Lustre: server umount lustre-MDT0000 complete [ 847.018252] LustreError: 29620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442370 with bad export cookie 17611793797905465354 [ 847.022149] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.026511] LustreError: 29620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 847.380154] Lustre: server umount lustre-MDT0001 complete [ 858.326260] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 860.792727] Key type lgssc unregistered [ 861.096649] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.100514] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.114103] LNet: Removed LNI 192.168.206.159@tcp [ 862.041195] Key type .llcrypt unregistered [ 862.043597] Key type ._llcrypt unregistered [ 885.058272] Key type ._llcrypt registered [ 885.060598] Key type .llcrypt registered [ 885.145795] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 886.292522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.308403] alg: No test for adler32 (adler32-zlib) [ 887.317886] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 887.521572] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 889.223148] Key type lgssc registered [ 890.195820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 897.260404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 897.274391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.681900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.202952] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 908.263908] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.294202] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 911.691149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.061614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.871257] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 924.273148] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 930.159338] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 936.711854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 936.869809] LustreError: 36813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.137241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.592824] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 943.242498] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 943.635700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 950.079979] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 963.769026] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:34:46 (1788442486) [ 965.436883] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 967.080576] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 973.470294] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:34:55 (1788442495) [ 981.450725] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:35:04 (1788442504) [ 982.381754] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 990.365331] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 990.389054] Lustre: Skipped 2 previous similar messages [ 991.804903] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:35:14 (1788442514) [ 995.006771] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 995.009751] Lustre: Skipped 3 previous similar messages [ 1012.432995] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1012.439454] Lustre: Skipped 6 previous similar messages [ 1015.335748] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:35:37 (1788442537) [ 1018.065264] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1021.305601] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1029.272801] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:35:51 (1788442551) [ 1031.000605] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1031.004332] Lustre: Skipped 3 previous similar messages [ 1043.852452] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:36:06 (1788442566) [ 1060.320746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1060.330632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.361332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.346331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.350531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.366769] Lustre: Skipped 1 previous similar message [ 1066.469687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1066.477855] Lustre: Skipped 1 previous similar message [ 1071.584287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.095244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1072.218530] Lustre: server umount lustre-OST0000 complete [ 1076.193370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1076.208099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.219658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.226225] Lustre: Skipped 1 previous similar message [ 1081.830676] LustreError: 35722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.862783] LustreError: 35722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1081.903374] Lustre: server umount lustre-MDT0000 complete [ 1085.665855] LustreError: 38557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442609 with bad export cookie 11803700970221813953 [ 1085.671477] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.678430] LustreError: 38557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1086.047707] Lustre: server umount lustre-MDT0001 complete [ 1096.776638] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1099.564623] Key type lgssc unregistered [ 1099.841817] LNet: 42858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.855603] LNetError: 42858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.875078] LNet: Removed LNI 192.168.206.159@tcp [ 1100.716120] Key type .llcrypt unregistered [ 1100.723070] Key type ._llcrypt unregistered [ 1118.248472] Key type ._llcrypt registered [ 1118.253632] Key type .llcrypt registered [ 1118.356396] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 1134.267410] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1136.233450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.329255] alg: No test for adler32 (adler32-zlib) [ 1137.661873] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1137.916383] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1139.655194] Key type lgssc registered [ 1140.952360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1174.751209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1174.782235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.217265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1176.276236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1176.371768] Lustre: lustre-MDT0000: new disk, initializing [ 1176.498932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.529264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1181.129912] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1190.584247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.646609] Lustre: 46013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1190.693423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1190.696918] Lustre: Skipped 1 previous similar message [ 1190.755688] Lustre: lustre-MDT0001: new disk, initializing [ 1190.830971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.858944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1190.874629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1195.278990] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1204.989556] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1210.699859] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1218.358513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1218.814384] Lustre: lustre-OST0000: new disk, initializing [ 1218.821895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1218.833478] Lustre: 47984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1218.934365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.931995] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1228.888463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1228.900872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1229.043911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1235.699782] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1240.638704] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1240.853880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1245.687851] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1246.014966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1249.760047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1249.766363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.781844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.794478] Lustre: Skipped 1 previous similar message [ 1254.382394] Lustre: server umount lustre-OST0000 complete [ 1260.004618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.010266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.017667] Lustre: Skipped 1 previous similar message [ 1260.031283] Lustre: Skipped 2 previous similar messages [ 1264.693774] Lustre: server umount lustre-MDT0000 complete [ 1268.352593] LustreError: 46821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442792 with bad export cookie 6806994410464742203 [ 1268.357860] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.367712] LustreError: 46821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1268.827431] Lustre: server umount lustre-MDT0001 complete [ 1277.839467] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:40:00 (1788442800) [ 1283.979457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.390291] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.417277] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1284.500086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.899891] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1289.702053] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.816242] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.340022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.746936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.741342] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1310.319711] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1316.171590] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.430206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.803524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.228871] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1337.057958] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1344.622919] LustreError: 52974:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1345.624554] Lustre: Modifying parameter general.timeout=1 in log params [ 1346.631912] Lustre: Modifying parameter general.timeout=2 in log params [ 1349.231395] Lustre: Modifying parameter general.timeout=5 in log params [ 1349.234387] Lustre: Skipped 2 previous similar messages [ 1353.967613] Lustre: Modifying parameter general.timeout=10 in log params [ 1353.974936] Lustre: Skipped 4 previous similar messages [ 1362.864675] Lustre: Modifying parameter general.timeout=19 in log params [ 1362.878473] Lustre: Skipped 8 previous similar messages [ 1379.101891] Lustre: Modifying parameter general.timeout=36 in log params [ 1379.105835] Lustre: Skipped 16 previous similar messages [ 1411.356879] Lustre: Modifying parameter general.timeout=73 in log params [ 1411.368273] Lustre: Skipped 36 previous similar messages [ 1471.061833] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:43:13 (1788442993) [ 1473.938569] Lustre: 59918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1475.867330] Lustre: 60016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1476.811641] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1476.820637] Lustre: Skipped 60 previous similar messages [ 1479.494909] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1479.503616] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1506.275218] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1506.283272] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1506.299812] LustreError: 61782:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1510.211802] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:43:52 (1788443032) [ 1522.152800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.167081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.194558] Lustre: Skipped 1 previous similar message [ 1522.217401] Lustre: Skipped 2 previous similar messages [ 1527.265471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.271324] Lustre: Skipped 2 previous similar messages [ 1528.514329] Lustre: server umount lustre-MDT0000 complete [ 1532.173829] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443056 with bad export cookie 6806994410464743085 [ 1532.187718] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.191532] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1532.385564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.386108] LustreError: 52287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.394792] Lustre: Skipped 1 previous similar message [ 1532.418176] LustreError: 52287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1532.547172] Lustre: server umount lustre-MDT0001 complete [ 1539.299537] Lustre: server umount lustre-OST0000 complete [ 1568.219053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.603260] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1569.115338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.580230] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1581.510519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.659738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1581.685841] Lustre: 64312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1587.070477] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1595.301071] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1601.344120] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1607.603149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1607.880525] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1608.337369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.352141] Lustre: Skipped 1 previous similar message [ 1613.968308] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1621.236862] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1622.754659] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1622.766210] Lustre: Skipped 19 previous similar messages [ 1622.818370] Lustre: 66878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1629.675603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.685299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.686136] Lustre: Skipped 1 previous similar message [ 1629.688954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1629.700775] Lustre: Skipped 1 previous similar message [ 1635.769371] Lustre: server umount lustre-OST0000 complete [ 1639.905121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.913781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.920841] Lustre: Skipped 1 previous similar message [ 1639.930599] Lustre: Skipped 3 previous similar messages [ 1647.649405] Lustre: server umount lustre-MDT0000 complete [ 1650.145749] LustreError: 64322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.161562] LustreError: 64322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1651.650682] LustreError: 64302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443175 with bad export cookie 6806994410464746088 [ 1651.655540] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.671136] LustreError: 64302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1652.136177] Lustre: server umount lustre-MDT0001 complete [ 1662.655720] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1665.195557] Key type lgssc unregistered [ 1665.492477] LNet: 68093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.515193] LNetError: 68093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.534929] LNet: Removed LNI 192.168.206.159@tcp [ 1667.409222] Key type .llcrypt unregistered [ 1667.414291] Key type ._llcrypt unregistered [ 1690.329229] Key type ._llcrypt registered [ 1690.331287] Key type .llcrypt registered [ 1690.509458] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1691.666699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.707831] alg: No test for adler32 (adler32-zlib) [ 1692.814633] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1693.136166] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1694.855193] Key type lgssc registered [ 1696.007851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.284867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.313814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.773266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.816845] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1714.149635] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.163512] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1716.111250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.581812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.835764] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1727.265368] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1732.067263] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1737.948288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1738.410440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.607693] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1750.757855] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1759.099251] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1767.904504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1767.913513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.926853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.470811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.479031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.505706] Lustre: Skipped 1 previous similar message [ 1773.426379] Lustre: server umount lustre-MDT0000 complete [ 1775.585681] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.608362] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1776.867069] LustreError: 72187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443300 with bad export cookie 3896577456166500339 [ 1776.875583] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1776.878095] LustreError: 72187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1777.379954] Lustre: server umount lustre-MDT0001 complete [ 1791.365787] Lustre: server umount lustre-OST0000 complete [ 1816.560501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.923137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1816.960384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1817.280418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.931988] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1830.393775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.484350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1830.519630] Lustre: 74379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1830.999654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.330956] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1842.495067] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1848.431182] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.975433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.225172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1855.583494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.645753] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1868.872495] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1870.589548] Lustre: 76952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1872.381207] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1879.520121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1879.523826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.566047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1879.578306] Lustre: Skipped 2 previous similar messages [ 1883.823378] Lustre: server umount lustre-OST0000 complete [ 1890.278714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.297857] Lustre: Skipped 1 previous similar message [ 1890.307238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.313277] Lustre: Skipped 3 previous similar messages [ 1893.706571] Lustre: server umount lustre-MDT0000 complete [ 1895.395539] LustreError: 74384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.423632] LustreError: 74384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1896.807498] LustreError: 76958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443420 with bad export cookie 3896577456166501186 [ 1896.811962] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.817606] LustreError: 76958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1897.174414] Lustre: server umount lustre-MDT0001 complete [ 1907.373350] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1910.461589] Key type lgssc unregistered [ 1910.778718] LNet: 78169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.788836] LNetError: 78169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.818854] LNet: Removed LNI 192.168.206.159@tcp [ 1911.769765] Key type .llcrypt unregistered [ 1911.774923] Key type ._llcrypt unregistered [ 1933.894211] Key type ._llcrypt registered [ 1933.896275] Key type .llcrypt registered [ 1933.964233] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1934.905913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.973254] alg: No test for adler32 (adler32-zlib) [ 1936.094674] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1936.331842] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1938.007173] Key type lgssc registered [ 1939.049738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.653754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1946.681335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.209803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.678797] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1958.368811] LustreError: 79583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.385349] LustreError: 79583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1960.274802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.897165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.802682] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1971.186079] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1975.839985] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1981.600863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.076201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.315889] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1995.829418] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.088095] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2002.963533] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2004.396771] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2004.407466] Lustre: Skipped 1 previous similar message [ 2006.897205] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2006.907115] Lustre: Skipped 2 previous similar messages [ 2011.549324] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2011.563530] Lustre: Skipped 4 previous similar messages [ 2019.647746] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2019.654702] Lustre: Skipped 8 previous similar messages [ 2036.086159] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2036.095448] Lustre: Skipped 16 previous similar messages [ 2068.670914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2068.676640] Lustre: Skipped 36 previous similar messages [ 2101.210567] Lustre: 87686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2104.602731] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:53:46 (1788443626) [ 2106.724577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2108.782184] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:53:51 (1788443631) [ 2116.580101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.580668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.605641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.621329] Lustre: Skipped 1 previous similar message [ 2119.908195] Lustre: server umount lustre-OST0000 complete [ 2124.257140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2124.264345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.273952] Lustre: Skipped 1 previous similar message [ 2124.281215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.823788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.825654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.935992] Lustre: server umount lustre-MDT0000 complete [ 2131.936393] LustreError: 79579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.953682] LustreError: 79579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2133.816563] LustreError: 79565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443657 with bad export cookie 6182939146145140669 [ 2133.823397] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.823850] LustreError: 79565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2134.240907] Lustre: server umount lustre-MDT0001 complete [ 2143.963293] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2146.278805] Key type lgssc unregistered [ 2146.564743] LNet: 88938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.573342] LNetError: 88938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.588069] LNet: Removed LNI 192.168.206.159@tcp [ 2147.501396] Key type .llcrypt unregistered [ 2147.503809] Key type ._llcrypt unregistered [ 2153.074177] Key type ._llcrypt registered [ 2153.078078] Key type .llcrypt registered [ 2153.356476] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2158.682319] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules [ 2160.041830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.123376] alg: No test for adler32 (adler32-zlib) [ 2161.501290] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2162.084331] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2163.968000] Key type lgssc registered [ 2165.655774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2167.513439] LustreError: 90189:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2185.413287] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2190.299685] LustreError: 90189:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2194.761167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2194.775423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.189285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.811442] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2206.691444] LustreError: 91213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.710591] LustreError: 91213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2208.855839] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:55:31 (1788443731) [ 2211.813413] LustreError: 91214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.025662] LustreError: 91245:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2213.043074] LustreError: 91245:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2213.052037] LustreError: 91245:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2213.306670] Lustre: server umount lustre-MDT0000 complete [ 2225.784223] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2228.445925] Key type lgssc unregistered [ 2228.682149] LNet: 92743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.689511] LNetError: 92743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.713366] LNet: Removed LNI 192.168.206.159@tcp [ 2229.547850] Key type .llcrypt unregistered [ 2229.550587] Key type ._llcrypt unregistered [ 2246.590490] Key type ._llcrypt registered [ 2246.594569] Key type .llcrypt registered [ 2246.718849] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2247.841870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.915735] alg: No test for adler32 (adler32-zlib) [ 2249.033081] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2249.322412] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2250.992523] Key type lgssc registered [ 2252.175094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.425697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2259.442986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.068703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.462488] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2270.181497] LustreError: 93961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.193613] LustreError: 93961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2273.041226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.747842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.253479] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2285.143847] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.550545] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2296.687848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.232287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.991566] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2309.853854] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2352.185460] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:57:54 (1788443874) [ 2353.846642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2355.308854] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:57:58 (1788443878) [ 2360.290512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2360.301893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.316525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.825938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.826145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.943165] Lustre: server umount lustre-MDT0000 complete [ 2366.946037] LustreError: 93962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.977013] LustreError: 93962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2369.670662] LustreError: 93942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443893 with bad export cookie 6544098158598770664 [ 2369.676909] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.685843] LustreError: 93942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2370.160568] Lustre: server umount lustre-MDT0001 complete [ 2384.209543] Lustre: server umount lustre-OST0000 complete [ 2394.381592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.705709] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.839205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.603138] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2400.225322] LustreError: 98238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.320792] LustreError: 98237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.162924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.825381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.976154] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2417.584406] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2423.048583] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2436.076739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2436.361801] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2436.373746] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2436.400688] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2436.414708] LustreError: 100320:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2436.434349] LustreError: 100320:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2436.464631] LustreError: 100320:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2436.542027] Lustre: server umount lustre-OST0000 complete [ 2436.546723] LustreError: 100320:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2447.232499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.554788] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2447.577760] LustreError: 98226:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2447.641752] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2447.658557] LustreError: 100788:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2447.667272] LustreError: 100788:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2447.679828] LustreError: 100788:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2447.739252] Lustre: server umount lustre-OST0000 complete [ 2447.743982] LustreError: 100788:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2457.799937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.954066] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2457.960619] Lustre: Found index 0 for lustre-OST0000, updating log [ 2457.969051] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2458.038748] Lustre: lustre-OST0000: new disk, initializing [ 2458.040800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2458.044413] Lustre: 101324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2458.511498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.056929] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2470.896182] Lustre: Failing over lustre-OST0000 [ 2471.106993] Lustre: server umount lustre-OST0000 complete [ 2477.217116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.223490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.237282] Lustre: Skipped 2 previous similar messages [ 2477.245188] Lustre: Skipped 2 previous similar messages [ 2482.657251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.662955] Lustre: Skipped 1 previous similar message [ 2486.818504] Lustre: server umount lustre-MDT0000 complete [ 2487.777283] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.804375] LustreError: 98232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2490.153474] LustreError: 98219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444014 with bad export cookie 6544098158598773408 [ 2490.165908] LustreError: 98219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2490.167091] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.559806] Lustre: server umount lustre-MDT0001 complete [ 2498.198065] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:00:20 (1788444020) [ 2512.901700] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2522.994058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.524797] LustreError: 103668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.694684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.630371] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2536.732266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.777916] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2545.654617] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2552.240335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.854421] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2559.170361] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2563.057732] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2563.071882] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2563.088443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2566.871096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.968791] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2567.051578] Lustre: lustre-OST0001: new disk, initializing [ 2567.054955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2567.065912] Lustre: 106000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2567.427566] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2567.471795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2567.503555] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2567.515767] Lustre: Skipped 2 previous similar messages [ 2568.508922] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2568.560665] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2568.563318] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2568.569916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2572.385690] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2577.142476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2577.165799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2577.276548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2580.299669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2586.009935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2590.484312] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.59@tcp (stopping) [ 2592.766816] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.794838] Lustre: Skipped 1 previous similar message [ 2596.819591] Lustre: server umount lustre-MDT0001 complete [ 2597.858778] LustreError: 104400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.875802] LustreError: 104400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2606.070440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.286780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2606.293164] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2606.370701] Lustre: 103657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2608.094443] LustreError: 107578:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2608.106979] LustreError: 107578:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2608.121470] LustreError: 103660:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2608.200352] LustreError: 107583:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.159@tcp: cfg command failed: rc = -17 [ 2608.208407] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2608.208407] [ 2611.039939] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2612.210454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2619.550138] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:02:22 (1788444142) [ 2634.907739] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2652.311639] Lustre: 109612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2673.109726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2676.324744] Lustre: 110748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2760.160804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2760.164753] LustreError: Skipped 1 previous similar message [ 2760.176260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.188709] Lustre: Skipped 1 previous similar message [ 2760.193862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.196323] Lustre: Skipped 4 previous similar messages [ 2765.805830] LustreError: 104687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.816756] LustreError: 104687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2765.912821] Lustre: server umount lustre-MDT0000 complete [ 2769.804525] LustreError: 103649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444293 with bad export cookie 6544098158598774409 [ 2769.808510] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.818328] LustreError: 103649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2770.054752] Lustre: server umount lustre-MDT0001 complete [ 2783.490337] Lustre: server umount lustre-OST0000 complete [ 2786.271271] Lustre: 93271:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444294/real 1788444294] req@ffff8b65c981df80 x1875319218066816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788444310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2786.300729] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.317809] Lustre: Skipped 3 previous similar messages [ 2787.167813] Lustre: server umount lustre-OST0001 complete [ 2792.506864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2802.434733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2812.968952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2822.984442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2840.940794] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2850.140192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.158314] Lustre: lustre-MDT0000: reset Object Index mappings [ 2850.715388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.719896] Lustre: Skipped 1 previous similar message [ 2855.182334] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2863.324581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.345964] Lustre: lustre-MDT0001: reset Object Index mappings [ 2869.267943] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2872.238698] Lustre: 115846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2878.208914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2884.991330] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2888.380974] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2893.026751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.771506] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2895.775095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2895.883763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2899.949818] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2907.092474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2910.343067] Lustre: 117724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2956.458183] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:07:59 (1788444479) [ 2982.905289] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.908077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.927585] Lustre: Skipped 3 previous similar messages [ 2982.943240] Lustre: Skipped 6 previous similar messages [ 2988.923474] Lustre: server umount lustre-MDT0000 complete [ 2992.747629] LustreError: 114679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444516 with bad export cookie 6544098158598821526 [ 2992.751252] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.770851] LustreError: 114679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2993.122023] LustreError: 114694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.154706] LustreError: 114694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2999.193666] Lustre: server umount lustre-MDT0001 complete [ 3009.229678] Lustre: server umount lustre-OST0000 complete [ 3018.722690] Lustre: server umount lustre-OST0001 complete [ 3023.959624] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 3032.832033] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3070.214463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.465587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3070.533063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3070.707283] Lustre: lustre-MDT0000: new disk, initializing [ 3070.866993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.875208] Lustre: Skipped 3 previous similar messages [ 3070.896445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3074.483749] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3081.221958] Lustre: server umount lustre-MDT0000 complete [ 3089.017489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.348443] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3089.359857] Lustre: 121993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3089.370988] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3095.072694] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3098.833975] Lustre: server umount lustre-MDT0000 complete [ 3107.173610] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:10:29 (1788444629) [ 3108.737534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3110.098259] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:10:32 (1788444632) [ 3125.259142] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3133.681733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.030141] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3145.667900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.793384] Lustre: 124041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3145.862399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3145.867481] Lustre: Skipped 1 previous similar message [ 3145.927573] Lustre: lustre-MDT0001: new disk, initializing [ 3146.037732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3146.052190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3150.246937] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3154.776346] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3160.848664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.158343] Lustre: lustre-OST0000: new disk, initializing [ 3161.161597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3161.169390] Lustre: 125616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3162.699747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3162.709623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3162.833223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3167.212240] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3177.311926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.507446] Lustre: lustre-OST0001: new disk, initializing [ 3177.515996] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3177.523285] Lustre: 126490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3179.294198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3179.308578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3179.445117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3184.177625] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3192.894217] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3195.728535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3537.280672] Lustre: 132902:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3537.289206] Lustre: 132902:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3537.295353] Lustre: 132902:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3537.301169] Lustre: 132902:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3537.307753] Lustre: 132902:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3537.315000] Lustre: 132902:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3805.456454] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:22:07 (1788445327) [ 3819.496079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.501583] Lustre: Skipped 4 previous similar messages [ 3819.504721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.508582] Lustre: Skipped 10 previous similar messages [ 3821.092606] Lustre: server umount lustre-MDT0000 complete [ 3823.903736] LustreError: 125615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445347 with bad export cookie 6544098158598888341 [ 3823.911386] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3823.918796] LustreError: 125615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3824.251356] Lustre: server umount lustre-MDT0001 complete [ 3840.991142] Lustre: 93272:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445348/real 1788445348] req@ffff8b65d06df800 x1875319226678016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3841.023603] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.048159] Lustre: Skipped 1 previous similar message [ 3844.127674] Lustre: 93272:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445352/real 1788445352] req@ffff8b65cbfde680 x1875319226678272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3849.185415] Lustre: 93273:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445357/real 1788445357] req@ffff8b66fe6d5c00 x1875319226678656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3849.202995] Lustre: 93273:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3852.256230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3852.402382] Lustre: server umount lustre-OST0000 complete [ 3854.303251] Lustre: 93271:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445362/real 1788445362] req@ffff8b65d06dd500 x1875319226678784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3861.810642] Lustre: server umount lustre-OST0001 complete [ 3868.431862] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 3878.145899] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3917.401686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.721240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3917.805343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3917.921634] Lustre: lustre-MDT0000: new disk, initializing [ 3918.067419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.073510] Lustre: Skipped 5 previous similar messages [ 3918.097040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3922.494344] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3933.811280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.965287] Lustre: 135951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3934.059853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3934.063883] Lustre: Skipped 1 previous similar message [ 3934.160246] Lustre: lustre-MDT0001: new disk, initializing [ 3934.372050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3934.385491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3939.263277] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3949.320399] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3955.955813] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3962.828547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.083177] Lustre: lustre-OST0000: new disk, initializing [ 3963.086764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.093175] Lustre: 137919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.511177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3964.530979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3964.667625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.035508] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3979.824025] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3988.740567] Lustre: lustre-MDD0000: changelog on [ 3991.801689] Lustre: lustre-MDD0001: changelog on [ 5647.445661] Lustre: lustre-MDD0001: changelog off [ 5649.052464] Lustre: 137081:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5649.778221] Lustre: lustre-MDD0000: changelog off [ 5659.624931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.626620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.637337] LustreError: Skipped 1 previous similar message [ 5659.647666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.653381] Lustre: Skipped 1 previous similar message [ 5660.856387] Lustre: server umount lustre-OST0000 complete [ 5664.736711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.742444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.750794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5664.756307] Lustre: Skipped 2 previous similar messages [ 5664.769371] Lustre: Skipped 1 previous similar message [ 5669.857588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.872507] Lustre: Skipped 1 previous similar message [ 5670.139575] Lustre: server umount lustre-MDT0000 complete [ 5673.768871] LustreError: 146252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447197 with bad export cookie 6544098158599708440 [ 5673.778199] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.795331] LustreError: 146252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5674.139769] Lustre: server umount lustre-MDT0001 complete [ 5684.235850] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 5686.511542] Key type lgssc unregistered [ 5686.804759] LNet: 147006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5686.815762] LNetError: 147006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5686.832533] LNet: Removed LNI 192.168.206.159@tcp [ 5687.482294] Key type .llcrypt unregistered [ 5687.488092] Key type ._llcrypt unregistered [ 5704.798636] Key type ._llcrypt registered [ 5704.803580] Key type .llcrypt registered [ 5704.930483] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 5718.588758] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5719.156857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.183995] alg: No test for adler32 (adler32-zlib) [ 5720.197987] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5720.408908] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 5722.055194] Key type lgssc registered [ 5723.127934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5752.245474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5752.261289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.529222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5753.549616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5753.613860] Lustre: lustre-MDT0000: new disk, initializing [ 5753.712295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.728816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5757.707837] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5765.582150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.678824] Lustre: 150212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5765.716961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5765.719960] Lustre: Skipped 1 previous similar message [ 5765.767306] Lustre: lustre-MDT0001: new disk, initializing [ 5765.844539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.880096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5765.893697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5769.386713] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5777.602226] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5783.066963] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5789.197307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.483916] Lustre: lustre-OST0000: new disk, initializing [ 5789.488834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5789.500703] Lustre: 152183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5789.571203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.700950] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5796.982333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5797.003421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5797.179538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5805.471924] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5811.674304] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_module obdecho/obdecho [ 5813.404319] LustreError: 153184:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5813.416494] LustreError: 153184:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5815.028135] LustreError: 153280:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5827.552416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5827.561598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.574271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5828.071712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.075253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5828.101176] Lustre: Skipped 1 previous similar message [ 5828.107220] Lustre: Skipped 1 previous similar message [ 5831.590919] Lustre: server umount lustre-MDT0000 complete [ 5835.416433] LustreError: 150206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447359 with bad export cookie 1250519142735852301 [ 5835.423785] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.430367] LustreError: 150206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5835.724536] Lustre: server umount lustre-MDT0001 complete [ 5849.594756] Lustre: server umount lustre-OST0000 complete [ 5869.672427] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 5880.311821] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5913.961177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.235687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5914.263945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5914.358614] Lustre: lustre-MDT0000: new disk, initializing [ 5914.421453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5914.447943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5917.757272] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5926.179511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.301891] Lustre: 156998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5926.363119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5926.371286] Lustre: Skipped 1 previous similar message [ 5926.445000] Lustre: lustre-MDT0001: new disk, initializing [ 5926.532426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.570359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5926.581839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5930.083373] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5937.602414] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5942.129680] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5947.684411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5947.963288] Lustre: lustre-OST0000: new disk, initializing [ 5947.970188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5947.977496] Lustre: 158965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5948.061779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.603370] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5955.113769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5955.120869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5955.214908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5961.834322] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5966.168267] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5966.315700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5971.099915] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5971.301685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5975.520967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.526973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.541330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.541339] Lustre: Skipped 1 previous similar message [ 5975.567199] Lustre: Skipped 1 previous similar message [ 5979.583116] Lustre: server umount lustre-OST0000 complete [ 5985.764319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.765264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.773098] Lustre: Skipped 1 previous similar message [ 5985.775889] Lustre: Skipped 1 previous similar message [ 5989.046295] Lustre: server umount lustre-MDT0000 complete [ 5990.881330] LustreError: 158290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.893859] LustreError: 158290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5992.124688] LustreError: 156989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447516 with bad export cookie 1250519142735853260 [ 5992.127940] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.134735] LustreError: 156989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5992.455547] Lustre: server umount lustre-MDT0001 complete [ 5995.896184] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:58:38 (1788447518) [ 5997.308833] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5998.993534] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:58:41 (1788447521) [ 6007.324164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.789981] LustreError: 161375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.883121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.908038] LustreError: 161376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.151710] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6018.017309] LustreError: 161375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6021.295898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.298951] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6033.087631] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6038.669389] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.418992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.861494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.866479] Lustre: Skipped 1 previous similar message [ 6052.582835] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6059.937683] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.682231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6067.693726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.706397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.709374] Lustre: Skipped 1 previous similar message [ 6072.039889] Lustre: server umount lustre-MDT0000 complete [ 6073.838971] LustreError: 161376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.860809] LustreError: 161376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6074.919904] LustreError: 161357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447598 with bad export cookie 1250519142735854142 [ 6074.928281] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6076.189655] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.59@tcp (stopping) [ 6076.205971] Lustre: Skipped 4 previous similar messages [ 6078.944829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.956618] Lustre: Skipped 2 previous similar messages [ 6081.332327] Lustre: server umount lustre-MDT0001 complete [ 6087.601032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6099.750462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.032398] LustreError: 165216:0:(llog.c:1655:llog_backup()) MGC192.168.206.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6100.041987] Lustre: 165216:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6101.816247] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6101.828757] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6119.905800] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.918296] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6152.952057] LustreError: 165223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.970590] LustreError: 165223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 6160.864779] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6160.869548] Lustre: 165216:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6201.854888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6206.092988] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6206.946458] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6213.624712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.184485] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.201451] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 6223.460930] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6224.881347] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6230.497961] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.087616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6237.228842] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6238.639655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6255.113977] Lustre: server umount lustre-MDT0000 complete [ 6255.585779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6255.596105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.616690] Lustre: Skipped 1 previous similar message [ 6258.907287] LustreError: 161356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447782 with bad export cookie 1250519142735860596 [ 6258.909825] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6258.927511] LustreError: 161356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6259.253725] Lustre: server umount lustre-MDT0001 complete [ 6273.530149] Lustre: server umount lustre-OST0000 complete [ 6298.016575] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 6309.033707] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6344.751879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.978965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6344.998397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6345.080253] Lustre: lustre-MDT0000: new disk, initializing [ 6345.142086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.154250] Lustre: Skipped 1 previous similar message [ 6345.181710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6349.775805] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6360.572838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.716747] Lustre: 170335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6360.973018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6366.091508] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6375.754602] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.955544] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6388.600514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6388.852561] Lustre: lustre-OST0000: new disk, initializing [ 6388.854995] Lustre: Skipped 1 previous similar message [ 6388.857755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6388.864442] Lustre: Skipped 2 previous similar messages [ 6388.867616] Lustre: 172305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6394.782104] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6396.988446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6397.001154] Lustre: Skipped 1 previous similar message [ 6397.013203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6397.127697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6403.980121] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6408.890934] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6409.157436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6414.323981] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6414.604251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6417.888623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.892512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6417.912149] LustreError: Skipped 1 previous similar message [ 6417.914449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.914456] Lustre: Skipped 2 previous similar messages [ 6417.942828] Lustre: Skipped 2 previous similar messages [ 6423.030765] Lustre: server umount lustre-OST0000 complete [ 6427.618549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6433.508822] Lustre: server umount lustre-MDT0000 complete [ 6437.351834] LustreError: 170343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.374968] LustreError: 170343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6437.406357] LustreError: 170328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447961 with bad export cookie 1250519142735877431 [ 6437.414238] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.416247] LustreError: 170328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.689824] Lustre: server umount lustre-MDT0001 complete [ 6446.498861] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:06:08 (1788447968) [ 6453.007903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.971406] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6465.762680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.467695] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6477.277064] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6482.748188] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6489.495978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.338130] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6504.789382] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6514.252757] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:07:16 (1788448036) [ 6522.854370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6522.884618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.894141] Lustre: Skipped 2 previous similar messages [ 6522.897027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.900960] Lustre: Skipped 6 previous similar messages [ 6526.509755] Lustre: server umount lustre-OST0000 complete [ 6545.375429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6545.646876] Lustre: server umount lustre-MDT0000 complete [ 6549.139596] LustreError: 174694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448073 with bad export cookie 1250519142735878313 [ 6549.146549] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.148806] LustreError: 174694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6549.469571] Lustre: server umount lustre-MDT0001 complete [ 6558.743432] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 6562.538530] Key type lgssc unregistered [ 6562.962776] LNet: 178445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6562.967409] LNetError: 178445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6562.980197] LNet: Removed LNI 192.168.206.159@tcp [ 6564.662702] Key type .llcrypt unregistered [ 6564.669509] Key type ._llcrypt unregistered [ 6588.257573] Key type ._llcrypt registered [ 6588.260496] Key type .llcrypt registered [ 6588.364087] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6589.223291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6589.238160] alg: No test for adler32 (adler32-zlib) [ 6590.337954] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6590.668852] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 6592.375201] Key type lgssc registered [ 6593.416854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6601.346366] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6601.374294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6616.991406] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6622.111456] LustreError: 179693:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.159@tcp: failed processing log, type 4: rc = -110 [ 6627.231475] LustreError: 179693:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6627.259444] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6627.267297] LustreError: MGC192.168.206.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6627.283647] LustreError: 179693:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6627.299664] LustreError: 179693:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6627.320411] LustreError: 179693:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6627.393482] Lustre: server umount lustre-OST0000 complete [ 6627.398942] LustreError: 179693:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6637.532201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.052357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.020532] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6648.289298] LustreError: 180192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6648.329643] LustreError: 180192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6653.413507] LustreError: 180191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.877793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6654.354325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6658.997741] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6666.335566] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6671.916937] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6678.785831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.017588] LustreError: 182007:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6679.028904] Lustre: 182007:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6679.222184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.488942] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6691.831173] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6693.859742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.861138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.872987] Lustre: Skipped 1 previous similar message [ 6698.978337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.993021] Lustre: Skipped 4 previous similar messages [ 6704.096756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6708.191165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6708.438198] Lustre: server umount lustre-MDT0000 complete [ 6709.216171] LustreError: 180191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.236303] LustreError: 180191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6711.956490] LustreError: 180173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448235 with bad export cookie 15498694744741157061 [ 6711.958186] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.966338] LustreError: 180173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6712.374296] Lustre: server umount lustre-MDT0001 complete [ 6726.590647] Lustre: server umount lustre-OST0000 complete [ 6736.854858] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 6748.364513] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6784.711489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.010400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6785.044365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6785.141523] Lustre: lustre-MDT0000: new disk, initializing [ 6785.256622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.277937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6790.141521] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6800.885816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.041230] Lustre: 185533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6801.107736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6801.111185] Lustre: Skipped 1 previous similar message [ 6801.165603] Lustre: lustre-MDT0001: new disk, initializing [ 6801.234165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6801.265192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6801.275150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6805.862691] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6815.299596] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6821.318403] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.938990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.196439] Lustre: lustre-OST0000: new disk, initializing [ 6828.199726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6828.205342] Lustre: 187500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6828.295789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.708647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6833.721271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6833.873996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6834.868458] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6844.652284] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6849.369257] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6849.568530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6854.728974] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6854.996441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6859.751912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.755187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.759418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.760374] Lustre: Skipped 2 previous similar messages [ 6859.773045] Lustre: Skipped 3 previous similar messages [ 6863.311719] Lustre: server umount lustre-OST0000 complete [ 6867.936365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6867.943159] LustreError: Skipped 1 previous similar message [ 6867.953531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.971179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6869.986222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.421327] Lustre: server umount lustre-MDT0000 complete [ 6875.104876] LustreError: 185545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.525599] LustreError: 185525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448401 with bad export cookie 15498694744741157782 [ 6877.539460] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.813530] Lustre: server umount lustre-MDT0001 complete [ 6887.265944] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:13:29 (1788448409) [ 6905.248643] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 6907.554343] Key type lgssc unregistered [ 6907.918626] LNet: 190452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6907.935740] LNetError: 190452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6907.956645] LNet: Removed LNI 192.168.206.159@tcp [ 6908.915920] Key type .llcrypt unregistered [ 6908.922895] Key type ._llcrypt unregistered [ 6929.711617] Key type ._llcrypt registered [ 6929.714652] Key type .llcrypt registered [ 6929.831673] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6931.256230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6931.292943] alg: No test for adler32 (adler32-zlib) [ 6932.642715] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6933.044321] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 6934.775206] Key type lgssc registered [ 6935.936903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6943.249031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6943.279286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.732982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.171853] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6954.976858] LustreError: 191672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.004669] LustreError: 191672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6956.362381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.720750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.950620] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6968.178546] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6973.634924] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6980.341429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.595731] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6980.598384] LustreError: 193539:0:(llog.c:1696:llog_backup()) MGC192.168.206.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6980.603947] Lustre: 193539:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6980.721701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.688767] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6992.077808] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6995.439083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.450940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6995.456039] Lustre: Skipped 2 previous similar messages [ 6995.469084] Lustre: Skipped 2 previous similar messages [ 7000.517957] Lustre: server umount lustre-MDT0000 complete [ 7000.552417] LustreError: 191673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.572789] LustreError: 191673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7003.484562] LustreError: 191654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448527 with bad export cookie 5765807455988161331 [ 7003.493725] LustreError: 191654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7003.493754] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.786933] Lustre: server umount lustre-MDT0001 complete [ 7016.314575] Lustre: server umount lustre-OST0000 complete [ 7024.602662] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 7034.355960] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7072.087519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.286307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7072.305065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7072.365922] Lustre: lustre-MDT0000: new disk, initializing [ 7072.497689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.511687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7076.883721] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7089.146483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.261163] Lustre: 197063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7089.300262] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7089.302415] Lustre: Skipped 1 previous similar message [ 7089.307438] LustreError: 197845:0:(llog.c:1696:llog_backup()) MGC192.168.206.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7089.325384] LustreError: 197845:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7089.331565] Lustre: 197845:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7089.360831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7089.375224] Lustre: Skipped 1 previous similar message [ 7089.490521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7089.501111] Lustre: lustre-MDT0001: new disk, initializing [ 7089.635852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7089.706205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7089.710273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7094.692314] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7104.344459] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7111.343040] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7119.972632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.412198] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7120.418317] Lustre: Skipped 1 previous similar message [ 7120.421426] LustreError: 199008:0:(llog.c:1696:llog_backup()) MGC192.168.206.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7120.427183] LustreError: 199008:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7120.437223] Lustre: 199008:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7120.443939] Lustre: 199008:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7120.489199] Lustre: lustre-OST0000: new disk, initializing [ 7120.497905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7120.518059] Lustre: 199031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.635572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.135841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7126.152397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7126.245873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7128.369759] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7139.068766] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7143.955480] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7144.214780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7149.204151] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7149.433890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7152.097250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7152.105243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.138611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.146117] Lustre: Skipped 1 previous similar message [ 7157.244580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.256776] Lustre: Skipped 1 previous similar message [ 7157.883375] Lustre: server umount lustre-OST0000 complete [ 7166.454235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7166.466955] LustreError: Skipped 1 previous similar message [ 7166.472624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.495451] Lustre: Skipped 1 previous similar message [ 7166.512395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7171.405589] Lustre: server umount lustre-MDT0000 complete [ 7172.578585] LustreError: 197877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.592254] LustreError: 197877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7175.567124] LustreError: 197054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448699 with bad export cookie 5765807455988162052 [ 7175.568406] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7175.586466] LustreError: 197054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7176.320221] Lustre: server umount lustre-MDT0001 complete [ 7185.021618] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:18:27 (1788448707) [ 7202.198692] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7211.042618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.363241] LustreError: 202157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7211.417522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.608320] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7216.608639] LustreError: 202158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.330766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.146165] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7232.214488] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7238.706319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.061507] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7253.938159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.272141] Lustre: lustre-OST0001: new disk, initializing [ 7254.283227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7254.294386] Lustre: 204464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7254.478135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7254.487902] Lustre: Skipped 2 previous similar messages [ 7261.221628] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7263.334100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7263.343833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7263.459434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7272.342937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7284.130564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7286.635943] Lustre: DEBUG MARKER: ADD OST3 [ 7292.353755] loop: module loaded [ 7292.445707] loop0: detected capacity change from 0 to 204800000 [ 7292.852108] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7292.876520] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7292.911941] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7299.795732] Lustre: DEBUG MARKER: STOP OST3 [ 7299.962535] loop0: detected capacity change from 0 to 204800000 [ 7306.408031] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.839821] Lustre: lustre-OST0003: new disk, initializing [ 7306.846893] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7306.858688] Lustre: 206543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7308.462638] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7308.577095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7308.588249] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7308.644597] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7309.156874] Lustre: Failing over lustre-OST0003 [ 7309.286470] Lustre: server umount lustre-OST0003 complete [ 7310.498059] LustreError: 206692:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7313.691718] LustreError: 203643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.712641] LustreError: 203643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7313.903113] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.923615] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.939066] Lustre: Skipped 2 previous similar messages [ 7314.290832] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7324.562605] Lustre: DEBUG MARKER: START OST3 again [ 7331.671310] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.003508] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7332.009845] Lustre: Skipped 1 previous similar message [ 7332.024226] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7333.070360] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7334.033921] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7334.035711] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7337.924302] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7357.478324] Lustre: Failing over lustre-OST0003 [ 7357.625773] Lustre: server umount lustre-OST0003 complete [ 7357.934834] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.960751] LustreError: 203644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.985811] LustreError: 203644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7378.402527] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.407303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7378.412838] Lustre: Skipped 1 previous similar message [ 7378.415121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.415130] Lustre: Skipped 2 previous similar messages [ 7381.654472] Lustre: server umount lustre-MDT0000 complete [ 7385.443759] LustreError: 205233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448909 with bad export cookie 5765807455988162934 [ 7385.460534] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.461297] LustreError: 205233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7390.691146] LustreError: 205253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.714354] LustreError: 205253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7391.925202] Lustre: server umount lustre-MDT0001 complete [ 7395.412292] Lustre: server umount lustre-OST0000 complete [ 7399.211365] Lustre: server umount lustre-OST0001 complete [ 7405.499872] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 7412.916258] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7449.703590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.973617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.034489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7450.133957] Lustre: lustre-MDT0000: new disk, initializing [ 7450.262415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7454.738608] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7463.531690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.671662] Lustre: 212074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7463.906885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.913087] Lustre: Skipped 1 previous similar message [ 7463.964445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7468.375212] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7477.290960] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7482.205566] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7488.366173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.588454] Lustre: lustre-OST0000: new disk, initializing [ 7488.593190] Lustre: Skipped 1 previous similar message [ 7488.599392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7488.602269] Lustre: Skipped 2 previous similar messages [ 7488.605297] Lustre: 214040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7490.048022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7490.055703] Lustre: Skipped 1 previous similar message [ 7490.068889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7490.197043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7494.187492] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7502.908639] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7507.936243] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7508.148357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7512.722450] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7512.956032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7515.622727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.624250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.633737] LustreError: Skipped 1 previous similar message [ 7515.635410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.635416] Lustre: Skipped 7 previous similar messages [ 7515.649052] Lustre: Skipped 6 previous similar messages [ 7521.358983] Lustre: server umount lustre-OST0000 complete [ 7525.357193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7531.655774] Lustre: server umount lustre-MDT0000 complete [ 7535.914842] LustreError: 212867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449059 with bad export cookie 5765807455988165223 [ 7535.920620] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.932016] LustreError: 212867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7536.352159] Lustre: server umount lustre-MDT0001 complete [ 7539.895074] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:24:22 (1788449062) [ 7555.953667] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 7566.516507] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7601.940118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.088287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7602.108663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7602.158135] Lustre: lustre-MDT0000: new disk, initializing [ 7602.273930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7605.380216] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7617.053359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.233849] Lustre: 218653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.706936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7623.160984] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7632.773328] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7639.260838] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7645.864091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.093930] Lustre: 220624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7647.757713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7651.625068] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7660.443413] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7664.468781] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7664.664510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7668.865816] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7669.042614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7673.312775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.315102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.329985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.335446] Lustre: Skipped 3 previous similar messages [ 7673.340302] Lustre: Skipped 7 previous similar messages [ 7685.087125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7685.239264] Lustre: server umount lustre-OST0000 complete [ 7695.173255] Lustre: server umount lustre-MDT0000 complete [ 7698.574653] LustreError: 218643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449222 with bad export cookie 5765807455988166112 [ 7698.580327] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7698.595251] LustreError: 218643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7698.975797] Lustre: server umount lustre-MDT0001 complete [ 7706.935033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.240764] LustreError: 222842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.280553] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7719.029125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.676584] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7729.395920] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7733.455347] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7739.818976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7740.215035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.222224] Lustre: Skipped 6 previous similar messages [ 7746.048934] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7752.650337] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7762.926733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7762.928882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.939695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7762.945836] Lustre: Skipped 3 previous similar messages [ 7762.971382] Lustre: Skipped 9 previous similar messages [ 7768.122738] Lustre: server umount lustre-OST0000 complete [ 7779.067773] Lustre: server umount lustre-MDT0000 complete [ 7783.086903] LustreError: 222822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449307 with bad export cookie 5765807455988166994 [ 7783.099895] LustreError: 222822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7783.102801] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7783.707737] Lustre: server umount lustre-MDT0001 complete [ 7890.971282] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 7894.366451] Key type lgssc unregistered [ 7894.570111] LNet: 226507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7894.576229] LNetError: 226507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.589746] LNet: Removed LNI 192.168.206.159@tcp [ 7895.294182] Key type .llcrypt unregistered [ 7895.297684] Key type ._llcrypt unregistered [ 7914.580274] Key type ._llcrypt registered [ 7914.583138] Key type .llcrypt registered [ 7914.694214] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 7929.027220] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7930.055255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7930.194534] alg: No test for adler32 (adler32-zlib) [ 7931.364548] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7931.589111] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 7933.263155] Key type lgssc registered [ 7934.306923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7969.276248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7969.310149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.647840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7970.722248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7970.813708] Lustre: lustre-MDT0000: new disk, initializing [ 7970.918286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.943915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7976.258427] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7985.837370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.941673] Lustre: 229860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7985.984360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7985.988764] Lustre: Skipped 1 previous similar message [ 7986.082405] Lustre: lustre-MDT0001: new disk, initializing [ 7986.224116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.263934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7986.287189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7990.286906] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7998.788553] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8004.063980] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8010.405374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8010.746162] Lustre: lustre-OST0000: new disk, initializing [ 8010.751309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8010.758933] Lustre: 231828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8010.851796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8016.221676] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8019.035628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8019.052030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8019.175347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8025.211215] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.095248] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8029.395905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8033.116360] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8033.292893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8039.904843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8039.906892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.918127] LustreError: Skipped 1 previous similar message [ 8039.934445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8039.935570] Lustre: Skipped 1 previous similar message [ 8039.951869] Lustre: Skipped 1 previous similar message [ 8041.372641] Lustre: server umount lustre-OST0000 complete [ 8045.026073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.035655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.047167] Lustre: Skipped 1 previous similar message [ 8050.145540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.153685] Lustre: Skipped 2 previous similar messages [ 8055.267146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.276313] Lustre: Skipped 1 previous similar message [ 8059.360161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8059.360603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.376524] Lustre: Skipped 1 previous similar message [ 8059.564373] Lustre: server umount lustre-MDT0000 complete [ 8063.252447] LustreError: 229853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449587 with bad export cookie 13477325939564340120 [ 8063.258864] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.268395] LustreError: 229853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8063.542726] Lustre: server umount lustre-MDT0001 complete [ 8081.238613] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8093.213961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8093.956088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8099.101282] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8099.296120] LustreError: 234761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.308874] LustreError: 234761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8103.396855] LustreError: 234762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.361450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.668251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8111.616281] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8114.608344] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8120.291039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.663943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.353785] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8134.170152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.332627] Lustre: lustre-OST0001: new disk, initializing [ 8134.337273] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8134.343043] Lustre: 237068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.237401] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8143.953890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8143.964213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8144.041806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8149.620931] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8156.528479] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8168.541909] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:34:50 (1788449690) [ 8174.561596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8174.572317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.588147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8176.640212] Lustre: server umount lustre-MDT0000 complete [ 8180.198836] LustreError: 234761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.227559] LustreError: 234761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8180.383649] LustreError: 237067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449704 with bad export cookie 13477325939564341002 [ 8180.388357] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8180.397855] LustreError: 237067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8180.859990] Lustre: server umount lustre-MDT0001 complete [ 8194.663804] Lustre: server umount lustre-OST0000 complete [ 8208.527783] Lustre: server umount lustre-OST0001 complete [ 8214.556835] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 8223.917844] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8258.535272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.736679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8258.754513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8258.818184] Lustre: lustre-MDT0000: new disk, initializing [ 8258.866911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.871539] Lustre: Skipped 1 previous similar message [ 8258.884304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8263.313617] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8272.650860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.767887] Lustre: 241016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8272.818326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8272.824070] Lustre: Skipped 1 previous similar message [ 8272.907921] Lustre: lustre-MDT0001: new disk, initializing [ 8273.022885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8273.038700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8276.685273] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8284.840726] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8289.730522] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8294.631528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8294.846269] Lustre: lustre-OST0000: new disk, initializing [ 8294.849432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8294.857138] Lustre: 242985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8294.940214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.948290] Lustre: Skipped 1 previous similar message [ 8295.941815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8295.962163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8296.041477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8300.509663] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8308.797907] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8313.444801] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8313.604102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8317.895920] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8318.099871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8321.513562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.525781] Lustre: Skipped 3 previous similar messages [ 8321.526116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8321.529655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8321.529661] Lustre: Skipped 4 previous similar messages [ 8334.304514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8334.533826] Lustre: server umount lustre-OST0000 complete [ 8339.425052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8339.440717] LustreError: Skipped 1 previous similar message [ 8339.448603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.468743] Lustre: Skipped 1 previous similar message [ 8344.881290] Lustre: server umount lustre-MDT0000 complete [ 8347.104635] LustreError: 241025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.127867] LustreError: 241025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8348.986697] LustreError: 241010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449872 with bad export cookie 13477325939564343137 [ 8348.998201] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.004030] LustreError: 241010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8349.016862] LustreError: 228086:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 8349.327558] Lustre: server umount lustre-MDT0001 complete [ 8358.082956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.614895] LustreError: 245201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.113582] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8368.121707] LustreError: 245201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.135795] LustreError: 245201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8371.985729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.345516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8372.356032] Lustre: Skipped 1 previous similar message [ 8377.512339] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8385.362409] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8391.631656] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.177312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8406.113766] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8414.120839] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8561.639293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8561.651481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.665481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8561.682942] Lustre: Skipped 2 previous similar messages [ 8561.694467] Lustre: Skipped 9 previous similar messages [ 8567.603544] Lustre: server umount lustre-OST0000 complete [ 8571.882848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8578.046039] Lustre: server umount lustre-MDT0000 complete [ 8581.634231] LustreError: 248025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450105 with bad export cookie 13477325939564344019 [ 8581.639172] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8581.643223] LustreError: 248025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8582.023704] Lustre: server umount lustre-MDT0001 complete [ 8684.810073] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 8688.517642] Key type lgssc unregistered [ 8689.019486] LNet: 248816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8689.026128] LNetError: 248816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.059625] LNet: Removed LNI 192.168.206.159@tcp [ 8690.230144] Key type .llcrypt unregistered [ 8690.231712] Key type ._llcrypt unregistered [ 8715.076545] Key type ._llcrypt registered [ 8715.079716] Key type .llcrypt registered [ 8715.191201] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 8731.999244] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8733.443818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8733.561759] alg: No test for adler32 (adler32-zlib) [ 8734.700316] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8735.034206] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 8736.712279] Key type lgssc registered [ 8738.081491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8786.830198] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8797.371727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8797.396234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.682267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8798.710911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8798.771637] Lustre: lustre-MDT0000: new disk, initializing [ 8798.835958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.855928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8804.132935] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8814.340398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.502296] Lustre: 252884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8814.575421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8814.579596] Lustre: Skipped 1 previous similar message [ 8814.702314] Lustre: lustre-MDT0001: new disk, initializing [ 8814.828068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8814.874173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8814.891268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.307725] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8823.624765] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8828.920350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.178599] Lustre: lustre-OST0000: new disk, initializing [ 8829.188563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8829.196781] Lustre: 254517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8829.311651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8830.294836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8830.306400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8830.482142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8835.208259] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8845.480138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8845.672494] Lustre: lustre-OST0001: new disk, initializing [ 8845.677309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8845.689935] Lustre: 255389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8845.876810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8852.065943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8852.082694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8852.180841] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8852.361646] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8863.161068] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8868.578256] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8881.634515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8881.640667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.662112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.183737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.184395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.198902] Lustre: Skipped 1 previous similar message [ 8884.922531] Lustre: server umount lustre-MDT0000 complete [ 8889.227154] LustreError: 253679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450413 with bad export cookie 3285459703540142727 [ 8889.228090] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.238176] LustreError: 253679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8889.598335] Lustre: server umount lustre-MDT0001 complete [ 8903.908416] Lustre: server umount lustre-OST0000 complete [ 8917.962811] Lustre: server umount lustre-OST0001 complete [ 8931.695105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8941.168665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8949.703481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8959.606944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8979.698992] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8989.233071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.260808] Lustre: lustre-MDT0000: reset Object Index mappings [ 8989.639288] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.670916] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8989.726178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.299732] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8994.787132] LustreError: 260401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.910056] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.952739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.976118] Lustre: lustre-MDT0001: reset Object Index mappings [ 9003.474853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.954720] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9011.855828] Lustre: 261542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9018.592387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9024.967829] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9028.075350] LustreError: 261897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.186939] LustreError: 261898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.915902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9034.161132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.166345] Lustre: Skipped 1 previous similar message [ 9035.197808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9035.248949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9041.736857] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9049.926673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9053.396123] Lustre: 263415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9059.812061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9059.823178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.836960] Lustre: Skipped 1 previous similar message [ 9059.840991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.849747] Lustre: Skipped 3 previous similar messages [ 9060.438967] Lustre: server umount lustre-MDT0000 complete [ 9060.841611] LustreError: 260401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.875775] LustreError: 260401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9063.625806] LustreError: 260382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450587 with bad export cookie 3285459703540149517 [ 9063.641900] LustreError: 260382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9063.643948] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.008424] Lustre: server umount lustre-MDT0001 complete [ 9078.203442] Lustre: server umount lustre-OST0000 complete [ 9081.312485] Lustre: 250392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450589/real 1788450589] req@ffff8b65c78c7100 x1875326017858560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9081.335659] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.354042] Lustre: Skipped 3 previous similar messages [ 9081.928691] Lustre: server umount lustre-OST0001 complete [ 9100.596726] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:50:22 (1788450622) [ 9117.030939] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 9127.034883] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9161.576843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.906279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9161.945692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9162.049042] Lustre: lustre-MDT0000: new disk, initializing [ 9162.105513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.133777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9165.892625] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9175.277916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.415308] Lustre: 267683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9175.504108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9175.507490] Lustre: Skipped 1 previous similar message [ 9175.643741] Lustre: lustre-MDT0001: new disk, initializing [ 9175.807425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9175.816818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9179.030967] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9190.771952] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9196.731720] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9204.261732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.643791] Lustre: lustre-OST0000: new disk, initializing [ 9204.651712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9204.663667] Lustre: 269654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9205.957195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9205.964663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9206.137214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9210.994708] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9220.691529] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9225.552654] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9225.799726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9230.439870] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9230.636635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9236.962793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.964529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9236.977075] Lustre: Skipped 1 previous similar message [ 9236.985844] Lustre: Skipped 1 previous similar message [ 9238.952434] Lustre: server umount lustre-OST0000 complete [ 9247.201078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9247.220895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9247.222482] Lustre: Skipped 1 previous similar message [ 9247.247569] Lustre: Skipped 1 previous similar message [ 9249.112352] Lustre: server umount lustre-MDT0000 complete [ 9252.328944] LustreError: 267690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9252.351348] LustreError: 267690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9253.146212] LustreError: 267676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450777 with bad export cookie 3285459703540151610 [ 9253.147442] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.158684] LustreError: 267676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9253.610469] Lustre: server umount lustre-MDT0001 complete [ 9272.585985] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9283.488288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.001044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.009789] Lustre: Skipped 2 previous similar messages [ 9289.134349] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9289.205197] LustreError: 272588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9289.243694] LustreError: 272588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9297.622544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.422096] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9306.697533] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9313.690948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.957534] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9330.065775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9330.255497] Lustre: lustre-OST0001: new disk, initializing [ 9330.259345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9330.265729] Lustre: 274896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9332.179094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9332.204336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9332.339641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9337.077321] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9346.741575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9350.613478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9355.472924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9355.500169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9362.940716] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.947167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9362.964686] Lustre: Skipped 3 previous similar messages [ 9362.988937] Lustre: Skipped 3 previous similar messages [ 9367.705830] Lustre: server umount lustre-MDT0000 complete [ 9368.035046] LustreError: 273317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.066637] LustreError: 273317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9371.567985] LustreError: 272568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450895 with bad export cookie 3285459703540152492 [ 9371.581429] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9372.174175] Lustre: server umount lustre-MDT0001 complete [ 9386.520100] Lustre: server umount lustre-OST0000 complete [ 9389.343775] Lustre: 250394:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450897/real 1788450897] req@ffff8b65d203fb80 x1875326017971712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9390.277220] Lustre: server umount lustre-OST0001 complete [ 9406.852443] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9417.457476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.849387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.855196] Lustre: Skipped 3 previous similar messages [ 9421.324995] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9428.546405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9433.119135] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9435.862053] Lustre: 279094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9442.479213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9450.075851] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9460.415377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9462.989121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9463.027608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9467.206843] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9474.672453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9478.578075] Lustre: 280969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9483.445732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9483.544797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9492.493779] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:56:54 (1788451014) [ 9509.347469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.354512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9509.360815] Lustre: Skipped 3 previous similar messages [ 9509.372253] Lustre: Skipped 1 previous similar message [ 9513.457842] Lustre: server umount lustre-MDT0000 complete [ 9514.473428] LustreError: 281568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.490506] LustreError: 281568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 9516.841381] LustreError: 280971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451040 with bad export cookie 3285459703540159275 [ 9516.847102] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.851355] LustreError: 280971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9531.359746] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9532.030646] Lustre: server umount lustre-MDT0001 complete [ 9543.488154] Lustre: server umount lustre-OST0000 complete [ 9554.011529] Lustre: server umount lustre-OST0001 complete [ 9560.196993] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 9571.015393] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9617.348250] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9626.032370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.290632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9626.334974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9626.419962] Lustre: lustre-MDT0000: new disk, initializing [ 9626.506293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9631.246596] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9641.517753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.620778] Lustre: 285210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9641.947907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9646.034899] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9655.303716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9655.617177] Lustre: 286843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9657.203678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9661.474629] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9671.993552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.204864] Lustre: 287712:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9674.218101] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9674.227535] Lustre: Skipped 1 previous similar message [ 9674.309584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9678.139895] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9688.599845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9692.144422] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9692.156378] Lustre: Skipped 1 previous similar message [ 9706.368743] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:00:28 (1788451228) [ 9708.517688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9708.525678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.541906] Lustre: Skipped 3 previous similar messages [ 9708.548931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9708.558064] Lustre: Skipped 8 previous similar messages [ 9710.595702] Lustre: server umount lustre-MDT0000 complete [ 9713.791421] LustreError: 286842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451237 with bad export cookie 3285459703540166702 [ 9713.795359] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9713.805809] LustreError: 286842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9714.153162] Lustre: server umount lustre-MDT0001 complete [ 9727.523830] Lustre: server umount lustre-OST0000 complete [ 9740.365806] Lustre: server umount lustre-OST0001 complete [ 9748.741118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.276466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.284313] Lustre: Skipped 7 previous similar messages [ 9753.096797] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9761.636609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.011983] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9772.989301] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9778.165434] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.727534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.221384] LustreError: 291802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.249406] LustreError: 291802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 9791.774129] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9798.994711] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9813.791662] loop1: detected capacity change from 0 to 204800000 [ 9814.017867] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9814.032168] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9814.044361] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9819.346267] loop1: detected capacity change from 0 to 204800000 [ 9824.886892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.989935] LustreError: 289963:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs7612: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9825.017682] LustreError: 289963:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs7612 [ 9825.028645] LustreError: 289963:0:(mgs_handler.c:517:mgs_target_reg()) fs7612-MDT0001: Error checking target index: rc = -13 [ 9825.043318] LustreError: fs7612-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9825.052713] LustreError: 293688:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9825.060803] LustreError: 293688:0:(tgt_mount.c:2129:server_put_super()) no obd fs7612-MDT0001 [ 9825.068642] LustreError: 293688:0:(tgt_mount.c:116:server_deregister_mount()) fs7612-MDT0001 not registered [ 9825.130859] Lustre: server umount fs7612-MDT0001 complete [ 9825.137453] LustreError: 293688:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9834.071732] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.173179] Lustre: Setting parameter fs7612-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7612-MDT0001 [ 9834.250409] Lustre: srv-fs7612-MDT0001: No data found on store. Initialize space. [ 9834.256629] Lustre: Skipped 4 previous similar messages [ 9834.374388] Lustre: fs7612-MDT0001: new disk, initializing [ 9834.377176] Lustre: Skipped 3 previous similar messages [ 9834.487341] Lustre: srv-fs7612-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9834.502467] Lustre: 294155:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7612-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9834.510605] LustreError: 294155:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7612-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9839.641636] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9845.815398] Lustre: server umount fs7612-MDT0001 complete [ 9856.945287] loop1: detected capacity change from 0 to 204800000 [ 9861.501908] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.007147] Lustre: 295721:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7612-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9862.018617] LustreError: 295721:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7612-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9866.746282] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9871.341398] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9872.126210] Lustre: server umount fs7612-MDT0001 complete [ 9876.504497] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9881.371261] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 9881.380195] Lustre: Skipped 4 previous similar messages [ 9881.573351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.591505] Lustre: Skipped 5 previous similar messages [ 9895.391253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9895.608485] Lustre: server umount lustre-MDT0000 complete [ 9903.079193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9903.284955] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.772068] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9908.724140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9914.701975] loop1: detected capacity change from 0 to 204800000 [ 9919.757601] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9920.271238] Lustre: 298171:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7612-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9920.282689] LustreError: 298171:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7612-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9924.165132] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9928.725718] Lustre: server umount fs7612-MDT0001 complete [ 9954.791077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9957.376831] Lustre: server umount lustre-OST0000 complete [ 9965.031946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9966.881701] Lustre: server umount lustre-MDT0000 complete [ 9970.673079] LustreError: 289957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451494 with bad export cookie 3285459703540169383 [ 9970.688589] LustreError: 289957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9970.688861] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9971.029199] Lustre: server umount lustre-MDT0001 complete [ 9983.486549] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 9986.004820] Key type lgssc unregistered [ 9986.357923] LNet: 300519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9986.376591] LNetError: 300519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9986.408079] LNet: Removed LNI 192.168.206.159@tcp [ 9987.372169] Key type .llcrypt unregistered [ 9987.374490] Key type ._llcrypt unregistered [10007.218749] Key type ._llcrypt registered [10007.222251] Key type .llcrypt registered [10007.350805] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10008.484110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10008.541124] alg: No test for adler32 (adler32-zlib) [10009.629541] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10009.948333] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10011.648092] Key type lgssc registered [10012.796646] Lustre: Echo OBD driver; http://www.lustre.org/ [10020.653533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10020.670739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.092214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.569611] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10031.590928] LustreError: 301793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.612617] LustreError: 301793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10034.375132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.814038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10039.003729] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10046.202341] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10051.364926] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10057.509615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.914317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10059.944307] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.257200] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10065.384178] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10069.928345] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10071.522982] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.551823] LustreError: 303623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10076.650278] LustreError: 303770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.689863] LustreError: 303770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10086.200315] loop1: detected capacity change from 0 to 204800000 [10086.381350] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10086.389169] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10086.401678] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10086.885857] LustreError: 303624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.902669] LustreError: 303624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10091.136459] loop1: detected capacity change from 0 to 204800000 [10096.211033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10096.380542] Lustre: fs31086-OST0000: new disk, initializing [10096.384975] Lustre: srv-fs31086-OST0000: No data found on store. Initialize space. [10096.392249] Lustre: 305527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10096.485714] Lustre: fs31086-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10102.784237] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10103.544418] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10103.560509] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10110.111942] Lustre: server umount fs31086-OST0000 complete [10120.915085] loop1: detected capacity change from 0 to 204800000 [10125.916607] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.066812] Lustre: fs31086-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10131.849459] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10136.036570] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.047177] LustreError: 303622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10140.119960] loop2: detected capacity change from 0 to 204800000 [10140.400154] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10140.421317] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10140.430557] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10141.193527] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10145.565130] loop2: detected capacity change from 0 to 204800000 [10146.285226] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10151.440509] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.542784] LustreError: fs31086-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10151.549156] LustreError: fs31086-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10151.563675] LustreError: 308566:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10151.574519] LustreError: 308566:0:(tgt_mount.c:2129:server_put_super()) no obd fs31086-OST0001 [10151.581692] LustreError: 308566:0:(tgt_mount.c:116:server_deregister_mount()) fs31086-OST0001 not registered [10151.624370] Lustre: server umount fs31086-OST0001 complete [10151.626852] LustreError: 308566:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10162.008207] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10162.209885] Lustre: fs31086-OST0001: new disk, initializing [10162.213543] Lustre: srv-fs31086-OST0001: No data found on store. Initialize space. [10162.225988] Lustre: 309116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10162.307770] Lustre: fs31086-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10168.990702] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10176.926897] Lustre: server umount fs31086-OST0000 complete [10185.060535] Lustre: server umount fs31086-OST0001 complete [10196.167312] loop1: detected capacity change from 0 to 204800000 [10200.827876] LustreError: 303770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.849676] LustreError: 303770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [10201.390217] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10201.639720] Lustre: fs31086-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10207.399722] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10214.226088] loop2: detected capacity change from 0 to 204800000 [10219.514879] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10226.051360] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10234.390426] Lustre: server umount fs31086-OST0000 complete [10242.810068] Lustre: server umount fs31086-OST0001 complete [10253.802045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.805327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10253.807946] Lustre: Skipped 1 previous similar message [10253.812443] Lustre: Skipped 1 previous similar message [10258.915444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10258.926670] Lustre: Skipped 1 previous similar message [10259.320466] Lustre: server umount lustre-OST0000 complete [10264.038699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.055927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.061814] Lustre: Skipped 1 previous similar message [10269.158341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10269.172463] Lustre: Skipped 1 previous similar message [10274.272819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.280741] Lustre: Skipped 1 previous similar message [10277.344849] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10277.458859] Lustre: server umount lustre-MDT0000 complete [10280.301351] LustreError: 309864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451804 with bad export cookie 15984777482010094071 [10280.308471] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.316956] LustreError: 309864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10280.548455] Lustre: server umount lustre-MDT0001 complete [10290.371171] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10293.624944] Key type lgssc unregistered [10293.926122] LNet: 315043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10293.934390] LNetError: 315043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10294.960357] LNet: Removed LNI 192.168.206.159@tcp [10295.844166] Key type .llcrypt unregistered [10295.848818] Key type ._llcrypt unregistered [10324.439995] Key type ._llcrypt registered [10324.444501] Key type .llcrypt registered [10324.569847] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [10338.983252] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10339.678218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10339.839991] alg: No test for adler32 (adler32-zlib) [10341.065240] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10341.288239] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10342.991150] Key type lgssc registered [10344.242463] Lustre: Echo OBD driver; http://www.lustre.org/ [10384.377935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10384.389743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.551213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10385.579275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10385.663270] Lustre: lustre-MDT0000: new disk, initializing [10385.734456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10385.750064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10389.632294] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10399.862149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.986796] Lustre: 318906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10400.025569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10400.028325] Lustre: Skipped 1 previous similar message [10400.090735] Lustre: lustre-MDT0001: new disk, initializing [10400.158238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10400.205244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10400.217830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10404.219941] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10412.898897] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10418.257865] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10424.394319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.697393] Lustre: lustre-OST0000: new disk, initializing [10424.701522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10424.709390] Lustre: 320874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10424.800421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10430.554500] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10434.096086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10434.102700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10434.210466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10437.801407] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10439.572947] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [10440.407500] Lustre: Unmounted lustre-client [10449.891548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10449.892172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.895882] LustreError: Skipped 1 previous similar message [10449.902849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10453.835307] Lustre: server umount lustre-OST0000 complete [10460.130570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10460.142252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10460.143566] Lustre: Skipped 2 previous similar messages [10460.156613] Lustre: Skipped 2 previous similar messages [10463.517116] Lustre: server umount lustre-MDT0000 complete [10466.938490] LustreError: 319699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451990 with bad export cookie 7147741577726244224 [10466.944836] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10466.947749] LustreError: 319699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10467.242269] Lustre: server umount lustre-MDT0001 complete [10476.360822] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10478.913606] Key type lgssc unregistered [10498.655408] LNet: 323034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10498.660714] LNetError: 323034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10569.322220] LNet: Removed LNI 192.168.206.159@tcp [10570.190440] Key type .llcrypt unregistered [10570.193250] Key type ._llcrypt unregistered [10585.370483] Key type ._llcrypt registered [10585.378326] Key type .llcrypt registered [10585.504362] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [10600.818575] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10601.579132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10601.711037] alg: No test for adler32 (adler32-zlib) [10602.707245] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10602.922512] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10604.559158] Key type lgssc registered [10605.533642] Lustre: Echo OBD driver; http://www.lustre.org/ [10631.008765] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:15:53 (1788452153) [10638.336794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10638.351700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.522168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10639.560890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10639.620321] Lustre: lustre-MDT0000: new disk, initializing [10639.739105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10639.760493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10644.979297] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10655.719091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.875376] Lustre: 326256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10655.946203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10655.957418] Lustre: Skipped 1 previous similar message [10656.055228] Lustre: lustre-MDT0001: new disk, initializing [10656.167150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10656.207496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10656.225126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10660.791587] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10669.103700] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10674.063624] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10680.390189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10680.612186] Lustre: lustre-OST0000: new disk, initializing [10680.614975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10680.619533] Lustre: 328227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10680.678081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10685.615918] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10688.077537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10688.091035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10688.206429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10694.504250] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10708.961313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10708.964069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10708.965469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10708.969434] LustreError: Skipped 1 previous similar message [10708.985078] Lustre: Skipped 1 previous similar message [10713.638830] Lustre: server umount lustre-OST0000 complete [10717.665202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.674821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.686315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10717.690759] Lustre: Skipped 1 previous similar message [10719.201778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.204057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10722.826514] Lustre: server umount lustre-MDT0000 complete [10726.974994] LustreError: 327053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452250 with bad export cookie 2284680627015916655 [10726.978945] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10726.984211] LustreError: 327053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10727.305548] Lustre: server umount lustre-MDT0001 complete [10739.586189] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10742.132087] Key type lgssc unregistered [10742.372583] LNet: 330136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10742.379279] LNetError: 330136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10742.404519] LNet: Removed LNI 192.168.206.159@tcp [10743.254249] Key type .llcrypt unregistered [10743.256500] Key type ._llcrypt unregistered [10762.238923] Key type ._llcrypt registered [10762.240150] Key type .llcrypt registered [10762.311549] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10762.969630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10762.986216] alg: No test for adler32 (adler32-zlib) [10764.118277] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10764.337785] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10766.008242] Key type lgssc registered [10767.054975] Lustre: Echo OBD driver; http://www.lustre.org/ [10773.160478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10773.181251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.562179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10778.417606] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10784.747485] LustreError: 331409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.767566] LustreError: 331409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10786.040323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.394458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10790.268300] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10796.573837] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10800.776098] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10806.802331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10807.165414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10812.542567] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10818.645963] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10828.840968] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10830.257832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10834.914340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.917638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10834.929118] Lustre: Skipped 1 previous similar message [10834.935318] Lustre: Skipped 2 previous similar messages [10836.750929] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.59@tcp (stopping) [10839.495425] Lustre: server umount lustre-MDT0000 complete [10840.032912] LustreError: 331409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.053901] LustreError: 331409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10841.845804] LustreError: 333503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.819465] LustreError: 333249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452366 with bad export cookie 14722363186197547879 [10842.829371] LustreError: 333249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10842.834644] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.153898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.156204] LustreError: 331406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10845.172151] Lustre: Skipped 1 previous similar message [10845.196307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10847.201456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10847.209580] Lustre: Skipped 1 previous similar message [10849.275917] Lustre: server umount lustre-MDT0001 complete [10853.295960] Lustre: server umount lustre-OST0000 complete [10859.992718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.614341] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10865.868501] Lustre: 335586:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.159@tcp [10867.464855] Lustre: server umount MGS complete [10873.748763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.057370] LustreError: 336157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.073461] LustreError: 336157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10874.114298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.189808] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10883.698125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10883.997936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10887.552622] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10893.948244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.201863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10898.718238] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10904.185676] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10905.300981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10909.794928] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10911.160971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10918.880589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10918.887181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.889511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10918.898773] Lustre: Skipped 1 previous similar message [10918.905240] Lustre: Skipped 1 previous similar message [10923.103899] Lustre: server umount lustre-OST0000 complete [10929.121742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10929.127544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.141168] Lustre: Skipped 1 previous similar message [10929.150214] Lustre: Skipped 1 previous similar message [10931.869095] Lustre: server umount lustre-MDT0000 complete [10934.246696] LustreError: 336158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.257729] LustreError: 336158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10934.397088] LustreError: 336138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452458 with bad export cookie 14722363186197548719 [10934.400269] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10934.409846] LustreError: 336138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10934.610484] Lustre: server umount lustre-MDT0001 complete [10942.134662] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10944.102349] Key type lgssc unregistered [10944.361093] LNet: 339592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10944.368406] LNetError: 339592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10944.380741] LNet: Removed LNI 192.168.206.159@tcp [10945.001092] Key type .llcrypt unregistered [10945.002778] Key type ._llcrypt unregistered [10959.508974] Key type ._llcrypt registered [10959.513290] Key type .llcrypt registered [10959.606047] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [10971.882749] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10972.541402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10972.658501] alg: No test for adler32 (adler32-zlib) [10973.629538] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10973.883520] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10975.527282] Key type lgssc registered [10976.300354] Lustre: Echo OBD driver; http://www.lustre.org/ [11006.474900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11006.537397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.717908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11007.739933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11007.813275] Lustre: lustre-MDT0000: new disk, initializing [11007.871709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11007.889783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11012.181179] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11021.728176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11021.811925] Lustre: 342734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11021.849530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11021.852681] Lustre: Skipped 1 previous similar message [11021.913773] Lustre: lustre-MDT0001: new disk, initializing [11021.993118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11022.036238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11022.045424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11025.911792] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11033.861846] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11039.249587] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11045.670590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11046.067830] Lustre: lustre-OST0000: new disk, initializing [11046.073249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11046.089021] Lustre: 344701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11046.166609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11051.916857] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11055.168502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11055.180434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11055.287153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11060.066486] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11064.774621] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11065.002520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11069.309465] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11069.474805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11076.064929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11076.068802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.080552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11076.089109] Lustre: Skipped 1 previous similar message [11081.190238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11081.200506] Lustre: Skipped 1 previous similar message [11085.279588] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11085.394677] Lustre: server umount lustre-OST0000 complete [11088.865954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11088.874036] LustreError: Skipped 1 previous similar message [11088.882857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11088.894094] Lustre: Skipped 1 previous similar message [11088.903702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.426342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.426770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.687284] Lustre: server umount lustre-MDT0000 complete [11097.513635] LustreError: 343528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452621 with bad export cookie 14900177977171730460 [11097.516639] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11097.527072] LustreError: 343528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11097.788836] Lustre: server umount lustre-MDT0001 complete [11104.399725] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:23:47 (1788452627) [11117.858619] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [11126.155975] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11153.194311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.360106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11153.382780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11153.456524] Lustre: lustre-MDT0000: new disk, initializing [11153.513443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11153.528908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11156.911710] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11165.566678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.661848] Lustre: 349455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11165.728443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11165.733681] Lustre: Skipped 1 previous similar message [11165.824503] Lustre: lustre-MDT0001: new disk, initializing [11165.929706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11165.983631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11166.007129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11169.886260] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11179.632940] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11185.926309] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11192.100743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11192.381968] Lustre: lustre-OST0000: new disk, initializing [11192.394042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11192.399700] Lustre: 351420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11192.555312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11198.000587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11198.012730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11198.160182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11198.498511] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11206.949215] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11244.513273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11244.516515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11244.519111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11244.533026] Lustre: Skipped 1 previous similar message [11244.537129] Lustre: Skipped 1 previous similar message [11254.754415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11254.756663] Lustre: Skipped 4 previous similar messages [11255.263804] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11255.405809] Lustre: server umount lustre-OST0000 complete [11259.872773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.885067] Lustre: Skipped 1 previous similar message [11264.838029] Lustre: server umount lustre-MDT0000 complete [11264.993521] LustreError: 352140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.010040] LustreError: 352140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11267.971149] LustreError: 349449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452791 with bad export cookie 14900177977171731349 [11267.977886] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11267.981308] LustreError: 349449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11268.232889] Lustre: server umount lustre-MDT0001 complete [11278.037913] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11280.876891] Key type lgssc unregistered [11281.136838] LNet: 353599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11281.149413] LNetError: 353599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11281.170975] LNet: Removed LNI 192.168.206.159@tcp [11281.852127] Key type .llcrypt unregistered [11281.856263] Key type ._llcrypt unregistered [11305.972387] Key type ._llcrypt registered [11305.974183] Key type .llcrypt registered [11306.068852] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11306.937345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11307.017763] alg: No test for adler32 (adler32-zlib) [11308.187783] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11308.397657] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11310.096097] Key type lgssc registered [11311.055436] Lustre: Echo OBD driver; http://www.lustre.org/ [11318.233848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11318.285362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11319.791085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11325.150028] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11330.021986] LustreError: 354894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11330.049677] LustreError: 354894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11334.359145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.648837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11338.768646] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11344.652438] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11348.428992] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11353.800614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11354.217542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11359.586853] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11365.002798] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11375.075941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11375.089227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.090499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11375.097979] Lustre: Skipped 1 previous similar message [11375.118567] Lustre: Skipped 1 previous similar message [11380.195434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11380.201487] Lustre: Skipped 1 previous similar message [11380.696911] Lustre: server umount lustre-OST0000 complete [11385.313348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11385.325235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11385.331574] Lustre: Skipped 1 previous similar message [11390.437965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11390.445708] Lustre: Skipped 1 previous similar message [11391.632217] Lustre: server umount lustre-MDT0000 complete [11395.571092] LustreError: 357382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.609436] LustreError: 357382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11395.748040] LustreError: 354876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452919 with bad export cookie 7280284137875242576 [11395.756542] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11395.768591] LustreError: 354876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11396.687534] Lustre: server umount lustre-MDT0001 complete [11408.151550] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11411.666543] Key type lgssc unregistered [11412.113076] LNet: 360607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11412.121282] LNetError: 360607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11412.143482] LNet: Removed LNI 192.168.206.159@tcp [11413.065156] Key type .llcrypt unregistered [11413.068462] Key type ._llcrypt unregistered [11427.710391] Key type ._llcrypt registered [11427.712518] Key type .llcrypt registered [11427.800274] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11428.526737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11428.591151] alg: No test for adler32 (adler32-zlib) [11429.584595] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11429.807103] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11431.471882] Key type lgssc registered [11432.822921] Lustre: Echo OBD driver; http://www.lustre.org/ [11441.008210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11441.034270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11442.538083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11447.038629] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11452.904984] LustreError: 364382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.916519] LustreError: 364382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11455.614036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.050748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11460.138696] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11466.889436] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11471.879599] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11479.753604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11480.367629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.483983] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11496.761742] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11505.590705] Lustre: server umount lustre-OST0000 complete [11512.295282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11512.312317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.343852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.314885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.316472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.339263] Lustre: Skipped 1 previous similar message [11518.435554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.452849] Lustre: Skipped 1 previous similar message [11519.803641] Lustre: server umount lustre-MDT0000 complete [11523.554017] Lustre: 362883:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453031/real 1788453031] req@000000001e3dcc2a x1875328843674112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788453047 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11523.559316] LustreError: 364375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.597805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.626746] LustreError: 364375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11523.960385] LustreError: 364306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453047 with bad export cookie 10425290208920475243 [11523.977692] LustreError: 364306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11523.980255] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11524.905567] Lustre: server umount lustre-MDT0001 complete [11536.395846] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11538.671862] Key type lgssc unregistered [11538.891928] LNet: 371803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11538.896171] LNetError: 371803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11538.910283] LNet: Removed LNI 192.168.206.159@tcp [11539.760115] Key type .llcrypt unregistered [11539.763251] Key type ._llcrypt unregistered [11555.001925] Key type ._llcrypt registered [11555.004824] Key type .llcrypt registered [11555.149271] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11556.606204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11556.665426] alg: No test for adler32 (adler32-zlib) [11557.829966] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11558.106841] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11559.808237] Key type lgssc registered [11561.254530] Lustre: Echo OBD driver; http://www.lustre.org/ [11568.951297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11568.981968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.542909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11575.036702] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11580.902531] LustreError: 375691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.918924] LustreError: 375691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11582.905911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.326074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11587.855532] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11595.261069] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11600.585270] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11607.517985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.844253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11613.349946] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11620.524864] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11629.538866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11629.554598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11629.581943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11631.588092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11631.598215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11631.601957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11636.705073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11636.716414] Lustre: Skipped 1 previous similar message [11641.835956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.841996] Lustre: Skipped 2 previous similar messages [11642.335275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11642.491820] Lustre: server umount lustre-OST0000 complete [11646.944784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.962105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.970827] Lustre: Skipped 1 previous similar message [11651.756146] Lustre: server umount lustre-MDT0000 complete [11652.068575] LustreError: 375681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.089151] LustreError: 375681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11655.977736] LustreError: 375612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453179 with bad export cookie 12145609136538119096 [11655.985674] LustreError: 375612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11655.987619] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11656.433747] Lustre: server umount lustre-MDT0001 complete [11672.472955] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11676.406070] Key type lgssc unregistered [11676.930099] LNet: 383661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11676.946256] LNetError: 383661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11677.997886] LNet: Removed LNI 192.168.206.159@tcp [11679.099106] Key type .llcrypt unregistered [11679.100751] Key type ._llcrypt unregistered [11698.374581] Key type ._llcrypt registered [11698.378709] Key type .llcrypt registered [11698.519694] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11699.825415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11699.894979] alg: No test for adler32 (adler32-zlib) [11700.963703] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11701.132690] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11702.768155] Key type lgssc registered [11704.457292] Lustre: Echo OBD driver; http://www.lustre.org/ [11712.660783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11712.679176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.074183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11718.337392] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11723.240891] LustreError: 387537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.258556] LustreError: 387537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11728.164674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.351800] LustreError: 387538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.806579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.463465] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11742.683975] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11749.348034] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11756.821203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11757.326242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11764.728673] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11772.237479] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11784.162682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11784.169475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.172156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.189457] Lustre: Skipped 1 previous similar message [11788.962685] Lustre: server umount lustre-OST0000 complete [11794.405621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.421158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.428232] Lustre: Skipped 2 previous similar messages [11798.747672] Lustre: server umount lustre-MDT0000 complete [11799.520476] LustreError: 387537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.544151] LustreError: 387537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11802.735350] LustreError: 392226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453326 with bad export cookie 13469467301732707480 [11802.746038] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11802.765459] LustreError: 392226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11803.172356] Lustre: server umount lustre-MDT0001 complete [11815.263923] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11818.789959] Key type lgssc unregistered [11819.086694] LNet: 393269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11819.092496] LNetError: 393269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11819.116690] LNet: Removed LNI 192.168.206.159@tcp [11820.100422] Key type .llcrypt unregistered [11820.102485] Key type ._llcrypt unregistered [11915.424864] Key type ._llcrypt registered [11915.427248] Key type .llcrypt registered [11915.526618] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11915.880970] Key type .llcrypt unregistered [11915.889367] Key type ._llcrypt unregistered [11990.945987] Key type ._llcrypt registered [11990.948232] Key type .llcrypt registered [11991.053406] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11991.422281] Key type .llcrypt unregistered [11991.425607] Key type ._llcrypt unregistered [12019.646578] Key type ._llcrypt registered [12019.650478] Key type .llcrypt registered [12019.782866] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12020.070553] Key type .llcrypt unregistered [12020.075134] Key type ._llcrypt unregistered [12047.451849] Key type ._llcrypt registered [12047.456671] Key type .llcrypt registered [12047.613594] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12047.971888] Key type .llcrypt unregistered [12047.978093] Key type ._llcrypt unregistered [12129.979159] Key type ._llcrypt registered [12129.985818] Key type .llcrypt registered [12130.150937] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12130.438426] Key type .llcrypt unregistered [12130.447928] Key type ._llcrypt unregistered [12162.558152] Key type ._llcrypt registered [12162.560755] Key type .llcrypt registered [12162.733941] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12163.051385] Key type .llcrypt unregistered [12163.053094] Key type ._llcrypt unregistered [12192.480440] Key type ._llcrypt registered [12192.483836] Key type .llcrypt registered [12192.576586] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12192.890318] Key type .llcrypt unregistered [12192.893507] Key type ._llcrypt unregistered [12223.621193] Key type ._llcrypt registered [12223.623990] Key type .llcrypt registered [12223.798743] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12224.041312] Key type .llcrypt unregistered [12224.043140] Key type ._llcrypt unregistered [12251.500350] Key type ._llcrypt registered [12251.502982] Key type .llcrypt registered [12251.631613] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12252.013845] Key type .llcrypt unregistered [12252.016149] Key type ._llcrypt unregistered [12281.411071] Key type ._llcrypt registered [12281.414319] Key type .llcrypt registered [12281.589977] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12281.934541] Key type .llcrypt unregistered [12281.938525] Key type ._llcrypt unregistered [12345.531704] Key type ._llcrypt registered [12345.536834] Key type .llcrypt registered [12345.639807] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12345.939277] Key type .llcrypt unregistered [12345.947128] Key type ._llcrypt unregistered [12374.247531] Key type ._llcrypt registered [12374.255225] Key type .llcrypt registered [12374.386909] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12374.755618] Key type .llcrypt unregistered [12374.760940] Key type ._llcrypt unregistered [12408.831617] Key type ._llcrypt registered [12408.837616] Key type .llcrypt registered [12408.978699] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12409.450306] Key type .llcrypt unregistered [12409.452827] Key type ._llcrypt unregistered [12460.410458] Key type ._llcrypt registered [12460.415339] Key type .llcrypt registered [12460.534328] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [12477.438516] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12479.303863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12479.338283] alg: No test for adler32 (adler32-zlib) [12480.476545] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12480.829990] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [12482.560329] Key type lgssc registered [12483.743482] Lustre: Echo OBD driver; http://www.lustre.org/ [12520.242351] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12520.279224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12521.581204] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12521.633646] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12521.728309] Lustre: tst250fs-MDT0000: new disk, initializing [12521.842056] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12521.858977] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12526.783238] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12537.626885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12537.751699] Lustre: 413830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12537.822510] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12537.830541] Lustre: Skipped 1 previous similar message [12537.937285] Lustre: tst250fs-MDT0001: new disk, initializing [12538.035891] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12538.088229] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12538.108480] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12542.274570] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12547.885696] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12554.981822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12555.245735] Lustre: tst250fs-OST0000: new disk, initializing [12555.251460] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12555.269033] Lustre: 415463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12555.392283] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12557.971469] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12557.986868] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12558.185843] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12561.451255] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12573.191619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12573.458776] Lustre: tst250fs-OST0001: new disk, initializing [12573.467749] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12573.481727] Lustre: 416339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12573.633987] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12579.461940] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12579.988249] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12580.007046] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12580.146505] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12585.401189] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12587.725678] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12587.735824] Lustre: Skipped 2 previous similar messages [12604.896538] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12604.917636] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.946024] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12605.922294] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.927341] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12605.936498] Lustre: Skipped 2 previous similar messages [12605.942972] Lustre: Skipped 2 previous similar messages [12610.904622] Lustre: server umount tst250fs-MDT0000 complete [12614.179715] LustreError: 415462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454138 with bad export cookie 2530897409929479912 [12614.198388] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12614.621108] Lustre: server umount tst250fs-MDT0001 complete [12628.229437] Lustre: server umount tst250fs-OST0000 complete [12642.070035] Lustre: server umount tst250fs-OST0001 complete [12663.436100] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [12673.558174] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12705.738136] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:50:28 (1788454228) [12707.489924] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12725.397299] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [12736.160174] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12766.739374] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12473 sec ========= 12:51:29 (1788454289) [12768.369870] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:51:30 (1788454290) === [12772.461310] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:51:34 (1788454294) === [12812.106668] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12814.966452] Key type lgssc unregistered [12815.292670] LNet: 425482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12815.299295] LNetError: 425482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12815.323688] LNet: Removed LNI 192.168.206.159@tcp [12815.998151] Key type .llcrypt unregistered [12816.009236] Key type ._llcrypt unregistered