[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 617303587 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003158] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.021324] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026730] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] *** VALIDATE tmpfs *** [ 0.032176] *** VALIDATE proc *** [ 0.033258] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041980] debug: unmapping init [mem 0xffffffffb8059000-0xffffffffb8060fff] [ 0.045187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054309] rcu: Hierarchical SRCU implementation. [ 0.056563] smp: Bringing up secondary CPUs ... [ 0.057615] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061149] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137419] node 0 deferred pages initialised in 70ms [ 0.139102] devtmpfs: initialized [ 0.140225] x86/mm: Memory block size: 128MB [ 0.142300] gcov: version magic: 0x41383552 [ 0.147346] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149296] pinctrl core: initialized pinctrl subsystem [ 0.150193] [ 0.150739] ************************************************************* [ 0.151020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152016] ** ** [ 0.153014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154012] ** ** [ 0.155019] ** This means that this kernel is built to expose internal ** [ 0.156014] ** IOMMU data structures, which may compromise security on ** [ 0.157015] ** your system. ** [ 0.158016] ** ** [ 0.159014] ** If you see this message and you are not debugging the ** [ 0.160014] ** kernel, report this immediately to your vendor! ** [ 0.161014] ** ** [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163017] ************************************************************* [ 0.164711] NET: Registered protocol family 16 [ 0.165431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168494] cpuidle: using governor menu [ 0.170780] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173733] PCI: Using configuration type 1 for base access [ 0.175103] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188056] cryptd: max_cpu_qlen set to 1000 [ 0.190283] ACPI: Added _OSI(Module Device) [ 0.191012] ACPI: Added _OSI(Processor Device) [ 0.192013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197683] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201250] ACPI: Interpreter enabled [ 0.202063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203011] ACPI: Using IOAPIC for interrupt routing [ 0.204103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214227] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219166] acpiphp: Slot [2] registered [ 0.220005] acpiphp: Slot [5] registered [ 0.220891] acpiphp: Slot [6] registered [ 0.221107] acpiphp: Slot [7] registered [ 0.222249] acpiphp: Slot [8] registered [ 0.223143] acpiphp: Slot [9] registered [ 0.224177] acpiphp: Slot [10] registered [ 0.225147] acpiphp: Slot [3] registered [ 0.226163] acpiphp: Slot [4] registered [ 0.227133] acpiphp: Slot [11] registered [ 0.228160] acpiphp: Slot [12] registered [ 0.229149] acpiphp: Slot [13] registered [ 0.230191] acpiphp: Slot [14] registered [ 0.231157] acpiphp: Slot [15] registered [ 0.232148] acpiphp: Slot [16] registered [ 0.233113] acpiphp: Slot [17] registered [ 0.234141] acpiphp: Slot [18] registered [ 0.235159] acpiphp: Slot [19] registered [ 0.236138] acpiphp: Slot [20] registered [ 0.237167] acpiphp: Slot [21] registered [ 0.238137] acpiphp: Slot [22] registered [ 0.239151] acpiphp: Slot [23] registered [ 0.240124] acpiphp: Slot [24] registered [ 0.241150] acpiphp: Slot [25] registered [ 0.242139] acpiphp: Slot [26] registered [ 0.243189] acpiphp: Slot [27] registered [ 0.244130] acpiphp: Slot [28] registered [ 0.245168] acpiphp: Slot [29] registered [ 0.246138] acpiphp: Slot [30] registered [ 0.247165] acpiphp: Slot [31] registered [ 0.248080] PCI host bridge to bus 0000:00 [ 0.249036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.250020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.254018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260901] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263739] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.277390] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.281011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.282039] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.284380] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.287568] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.290049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.293795] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.299018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.312021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.315801] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.332020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.344020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.364020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.378949] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.404025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.457000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.484020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.494158] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.514020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.528023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.539539] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.547021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.554019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.573021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.583238] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.627026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.640879] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.643439] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.646597] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.649399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.651277] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.657134] iommu: Default domain type: Passthrough [ 0.659520] SCSI subsystem initialized [ 0.661181] ACPI: bus type USB registered [ 0.663133] usbcore: registered new interface driver usbfs [ 0.665110] usbcore: registered new interface driver hub [ 0.667083] usbcore: registered new device driver usb [ 0.669169] pps_core: LinuxPPS API ver. 1 registered [ 0.671014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675063] PTP clock support registered [ 0.677048] EDAC MC: Ver: 3.0.0 [ 0.678643] PCI: Using ACPI for IRQ routing [ 0.680785] NetLabel: Initializing [ 0.682011] NetLabel: domain hash size = 128 [ 0.683007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684087] NetLabel: unlabeled traffic allowed by default [ 0.685238] vgaarb: loaded [ 0.687006] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.695000] clocksource: Switched to clocksource kvm-clock [ 0.780173] VFS: Disk quotas dquot_6.6.0 [ 0.781666] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.784265] *** VALIDATE ramfs *** [ 0.785567] *** VALIDATE hugetlbfs *** [ 0.788084] pnp: PnP ACPI init [ 0.790453] pnp: PnP ACPI: found 6 devices [ 0.805997] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.808854] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.810931] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.812826] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.815448] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.817387] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.820567] NET: Registered protocol family 2 [ 0.822838] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.826714] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.829278] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.834467] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.837635] TCP: Hash tables configured (established 65536 bind 65536) [ 0.840523] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.843065] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.845022] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.847232] NET: Registered protocol family 1 [ 0.849032] RPC: Registered named UNIX socket transport module. [ 0.850581] RPC: Registered udp transport module. [ 0.851610] RPC: Registered tcp transport module. [ 0.852719] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.854700] NET: Registered protocol family 44 [ 0.856572] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.858763] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.861035] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.863427] PCI: CLS 0 bytes, default 64 [ 0.865582] Unpacking initramfs... [ 2.325393] debug: unmapping init [mem 0xffff9bfbfcc54000-0xffff9bfbfffbffff] [ 2.331750] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334069] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.338438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.847509] Initialise system trusted keyrings [ 2.849442] Key type blacklist registered [ 2.851602] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.862033] zbud: loaded [ 2.865337] *** VALIDATE nfs *** [ 2.866880] *** VALIDATE nfs4 *** [ 2.868777] pstore: using deflate compression [ 2.872666] Platform Keyring initialized [ 2.993757] NET: Registered protocol family 38 [ 2.995845] Key type asymmetric registered [ 2.997551] Asymmetric key parser 'x509' registered [ 2.999698] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003131] io scheduler mq-deadline registered [ 3.005044] io scheduler kyber registered [ 3.007109] io scheduler bfq registered [ 3.009154] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.012650] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.015765] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.019137] ACPI: Power Button [PWRF] [ 3.024894] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.032477] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.061560] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.074346] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.102052] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.131689] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.160144] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.165425] Non-volatile memory driver v1.3 [ 3.167595] Linux agpgart interface v0.103 [ 3.210632] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.215493] vda: detected capacity change from 0 to 74711040 [ 3.232258] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.237712] vdb: detected capacity change from 0 to 1073741824 [ 3.254087] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.258108] vdc: detected capacity change from 0 to 2621440000 [ 3.273060] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275407] vdd: detected capacity change from 0 to 2621440000 [ 3.290747] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.293953] vde: detected capacity change from 0 to 4294967296 [ 3.307363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.309933] vdf: detected capacity change from 0 to 4294967296 [ 3.318772] libphy: Fixed MDIO Bus: probed [ 3.324976] usbcore: registered new interface driver usbserial_generic [ 3.327754] usbserial: USB Serial support registered for generic [ 3.331208] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.336341] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.338290] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.341384] mousedev: PS/2 mouse device common for all mice [ 3.344218] rtc_cmos 00:05: RTC can wake from S4 [ 3.347202] rtc_cmos 00:05: registered as rtc0 [ 3.348800] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.348869] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.351137] intel_pstate: CPU model not supported [ 3.357587] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.360352] hid: raw HID events driver (C) Jiri Kosina [ 3.362540] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.363643] usbcore: registered new interface driver usbhid [ 3.367662] usbhid: USB HID core driver [ 3.371265] drop_monitor: Initializing network drop monitor service [ 3.373995] Initializing XFRM netlink socket [ 3.376373] NET: Registered protocol family 10 [ 3.381366] Segment Routing with IPv6 [ 3.382814] NET: Registered protocol family 17 [ 3.385139] mpls_gso: MPLS GSO support [ 3.389873] RAS: Correctable Errors collector initialized. [ 3.391674] AVX version of gcm_enc/dec engaged. [ 3.393387] AES CTR mode by8 optimization enabled [ 3.479327] sched_clock: Marking stable (3479297760, 0)->(4470670094, -991372334) [ 3.483168] registered taskstats version 1 [ 3.485857] Loading compiled-in X.509 certificates [ 3.490445] zswap: loaded using pool lzo/zbud [ 3.514840] Key type big_key registered [ 3.525478] Key type encrypted registered [ 3.526809] ima: No TPM chip found, activating TPM-bypass! [ 3.529082] ima: Allocated hash algorithm: sha1 [ 3.530696] ima: No architecture policies found [ 3.532571] evm: Initialising EVM extended attributes: [ 3.534523] evm: security.selinux [ 3.535687] evm: security.ima [ 3.536768] evm: security.capability [ 3.538233] evm: HMAC attrs: 0x1 [ 3.540865] rtc_cmos 00:05: setting system clock to 2026-08-14 22:43:59 UTC (1786747439) [ 3.546676] debug: unmapping init [mem 0xffffffffb9003000-0xffffffffb91fffff] [ 3.549618] debug: unmapping init [mem 0xffffffffb7d82000-0xffffffffb8058fff] [ 3.563094] Write protecting the kernel read-only data: 28672k [ 3.566514] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.569725] debug: unmapping init [mem 0xffffffffb6d14000-0xffffffffb6dfffff] [ 3.606054] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.614864] systemd[1]: Detected virtualization kvm. [ 3.616866] systemd[1]: Detected architecture x86-64. [ 3.618903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.644665] systemd[1]: No hostname configured. [ 3.646133] systemd[1]: Set hostname to . [ 3.648363] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651182] systemd[1]: Initializing machine ID from random generator. [ 3.693894] random: ln: uninitialized urandom read (6 bytes read) [ 3.800358] random: systemd: uninitialized urandom read (16 bytes read) [ 3.803780] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.808974] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.813248] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.393207] device-mapper: uevent: version 1.0.3 [ 4.395441] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.220931] virtio_net virtio0 ens2: renamed from eth0 [ 5.222610] random: fast init done [ 5.289916] scsi host0: ata_piix [ 5.331346] scsi host1: ata_piix [ 5.332991] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.335849] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.869807] dracut-initqueue[602]: RTNETLINK answers: File exists [ 10.045776] random: crng init done [ 10.052118] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.452608] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.949937] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.991440] SELinux: Disabled at runtime. [ 16.105836] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.124715] systemd[1]: Detected virtualization kvm. [ 16.130480] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.357522] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.363110] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.377325] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.382833] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.388000] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.408142] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.422602] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 18.488287] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.115551] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.373756] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.550698] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.943431] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.033027] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.200584] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.102405] NFS: Registering the id_resolver key type [ 28.107752] Key type id_resolver registered [ 28.115290] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg110-server login: [ 83.822809] libcfs: loading out-of-tree module taints kernel. [ 83.883366] Key type ._llcrypt registered [ 83.885219] Key type .llcrypt registered [ 83.960970] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 103.665086] hrtimer: interrupt took 4572627 ns [ 105.863594] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 107.802974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.823080] alg: No test for adler32 (adler32-zlib) [ 109.175938] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 109.741763] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 111.463144] Key type lgssc registered [ 113.169380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.729235] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 183.058516] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 196.206115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.257709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.849757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.902511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.094258] Lustre: lustre-MDT0000: new disk, initializing [ 198.170928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.186202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.881853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.355258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.454169] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.495621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.500755] Lustre: Skipped 1 previous similar message [ 214.600468] Lustre: lustre-MDT0001: new disk, initializing [ 214.723835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.754297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.771823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.420601] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.193386] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.846467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.047215] Lustre: lustre-OST0000: new disk, initializing [ 234.050199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.054402] Lustre: 8466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.109965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.231194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.248449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.362162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.992646] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.997824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.256438] Lustre: lustre-OST0001: new disk, initializing [ 259.259329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.268279] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.403087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.064775] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.966456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.986071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.030561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.016665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.565887] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.781356] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.173615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing yml_node [ 301.881593] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 304.225437] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 307.041604] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 309.359465] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 18:49:03 EDT 2026 [ 328.951342] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 330.359241] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.080206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.093728] Lustre: Skipped 2 previous similar messages [ 346.095749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.103477] Lustre: Skipped 3 previous similar messages [ 351.221267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.227644] Lustre: Skipped 3 previous similar messages [ 351.586171] Lustre: server umount lustre-MDT0000 complete [ 355.503887] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747791 with bad export cookie 14630403877212203506 [ 355.514038] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.520337] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.325467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.327839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.353443] Lustre: Skipped 1 previous similar message [ 356.373995] Lustre: Skipped 1 previous similar message [ 360.352644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.356677] Lustre: Skipped 1 previous similar message [ 362.129589] Lustre: server umount lustre-MDT0001 complete [ 366.298773] Lustre: server umount lustre-OST0000 complete [ 370.514152] Lustre: server umount lustre-OST0001 complete [ 377.534799] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 386.315953] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 422.182534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.397337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.460375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.537265] Lustre: lustre-MDT0000: new disk, initializing [ 422.612244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.629478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.435419] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 436.860750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.006823] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.056749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.060435] Lustre: Skipped 1 previous similar message [ 437.167297] Lustre: lustre-MDT0001: new disk, initializing [ 437.302121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.332614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.347318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.517110] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 451.693752] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.723090] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.836988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.146303] Lustre: lustre-OST0000: new disk, initializing [ 465.151333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.159810] Lustre: 17063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.262577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.265922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.288691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.399916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.094270] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 482.457765] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.296811] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.568229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.618791] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.843550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.121678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.136046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.150270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.155652] Lustre: Skipped 1 previous similar message [ 507.360627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.389650] Lustre: Skipped 3 previous similar messages [ 509.407154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 509.710820] Lustre: server umount lustre-OST0000 complete [ 514.016641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.031402] LustreError: Skipped 1 previous similar message [ 514.033090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.039891] Lustre: Skipped 1 previous similar message [ 527.327234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.778473] Lustre: server umount lustre-MDT0000 complete [ 527.870335] LustreError: 17072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.903391] LustreError: 17072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 531.309787] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747967 with bad export cookie 14630403877212205039 [ 531.317224] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 531.328285] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 531.835278] Lustre: server umount lustre-MDT0001 complete [ 538.244605] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:52:52 (1786747972) [ 561.853333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.342094] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.472402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.671429] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 567.781145] LustreError: 19942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.899071] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.291616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.991951] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 591.103049] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 596.407319] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.572954] Lustre: Failing over lustre-MDT0000 [ 598.880951] Lustre: server umount lustre-MDT0000 complete [ 601.699814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.711957] Lustre: Skipped 1 previous similar message [ 601.718558] LustreError: 21093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.749122] LustreError: 21093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 612.324897] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.350136] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 617.445446] Lustre: 3659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748037/real 1786748037] req@ffff9bfb42582300 x1873540593908224/t0(0) o400->MGC192.168.201.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786748053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.485508] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.065657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.720708] LustreError: 21093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.737823] LustreError: 21093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 629.022542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.028684] Lustre: Skipped 1 previous similar message [ 629.075740] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 633.312225] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 634.337259] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 634.344570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.394621] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.158700] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 649.696249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.703937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.717704] Lustre: Skipped 1 previous similar message [ 649.722358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.730443] Lustre: Skipped 5 previous similar messages [ 653.186978] Lustre: server umount lustre-MDT0000 complete [ 657.314545] LustreError: 19923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748093 with bad export cookie 14630403877212206453 [ 657.321122] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.323969] LustreError: 19923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 657.635310] Lustre: server umount lustre-MDT0001 complete [ 665.903685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.240856] LustreError: 23289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.263541] LustreError: 23289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 670.670476] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 680.719039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.156991] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 693.802397] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 699.832712] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.865174] Lustre: Failing over lustre-MDT0000 [ 701.922979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.933776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.952787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.966530] Lustre: Skipped 2 previous similar messages [ 701.991207] Lustre: Skipped 3 previous similar messages [ 702.223798] Lustre: server umount lustre-MDT0000 complete [ 721.471811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.614934] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.885735] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 725.495116] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 726.953879] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 726.966282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 726.974387] Lustre: Skipped 1 previous similar message [ 727.000630] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 734.794900] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 741.350335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.382972] Lustre: Skipped 1 previous similar message [ 745.707976] Lustre: server umount lustre-MDT0000 complete [ 746.470062] LustreError: 23290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.490929] LustreError: 23290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 748.914218] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748184 with bad export cookie 14630403877212207391 [ 748.925377] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 748.927296] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.427233] Lustre: server umount lustre-MDT0001 complete [ 758.164726] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:56:32 (1786748192) [ 776.395877] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 788.330114] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 827.237166] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 841.249738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.691258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 841.752305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 841.917852] Lustre: lustre-MDT0000: new disk, initializing [ 842.078621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.081210] Lustre: Skipped 3 previous similar messages [ 842.131659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 847.683055] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 859.981596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.150206] Lustre: 29666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 860.534306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 865.016767] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 875.467779] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 882.067073] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 888.708564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 889.075821] Lustre: 31637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 890.719824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 890.781259] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 890.898124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 890.926582] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 896.001565] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 906.106642] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 912.552405] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 931.812737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.817710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.823862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.831994] Lustre: Skipped 1 previous similar message [ 931.843657] Lustre: Skipped 2 previous similar messages [ 945.120228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 945.647882] Lustre: server umount lustre-OST0000 complete [ 956.191627] Lustre: server umount lustre-MDT0000 complete [ 957.409203] LustreError: 31279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.428861] LustreError: 31279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 961.556988] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748397 with bad export cookie 14630403877212207790 [ 961.564672] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.572721] LustreError: 29658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 962.264424] Lustre: server umount lustre-MDT0001 complete [ 975.853720] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 978.842555] Key type lgssc unregistered [ 979.211561] LNet: 33623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.220622] LNetError: 33623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.241501] LNet: Removed LNI 192.168.201.110@tcp [ 979.972316] Key type .llcrypt unregistered [ 979.974589] Key type ._llcrypt unregistered [ 1008.212969] Key type ._llcrypt registered [ 1008.225293] Key type .llcrypt registered [ 1008.339240] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1009.093715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.123516] alg: No test for adler32 (adler32-zlib) [ 1010.352091] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1010.645308] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1012.311337] Key type lgssc registered [ 1013.190199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.726665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.737341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.275524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.783602] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1033.700938] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.716601] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1038.009255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.374760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1043.129466] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1050.817159] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1057.237070] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1063.732443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.911282] LustreError: 36888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.049240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.148800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1068.179603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1069.627377] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1076.085460] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.838436] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:02:03 (1786748523) [ 1090.991842] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1092.748220] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1099.450120] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:02:13 (1786748533) [ 1108.711665] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:02:23 (1786748543) [ 1109.661727] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1118.585888] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1118.594272] Lustre: Skipped 2 previous similar messages [ 1120.533458] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:02:35 (1786748555) [ 1123.777899] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1123.781802] Lustre: Skipped 3 previous similar messages [ 1139.373951] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1139.378720] Lustre: Skipped 6 previous similar messages [ 1142.186350] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:02:56 (1786748576) [ 1144.857341] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1148.077255] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1155.381537] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:03:10 (1786748590) [ 1157.000642] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1157.005841] Lustre: Skipped 3 previous similar messages [ 1169.658721] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:03:24 (1786748604) [ 1186.277569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1186.284189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.285781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.298352] LustreError: Skipped 1 previous similar message [ 1186.332992] Lustre: Skipped 1 previous similar message [ 1189.371618] Lustre: server umount lustre-OST0000 complete [ 1196.515495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.555057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.557620] Lustre: Skipped 1 previous similar message [ 1199.710822] Lustre: server umount lustre-MDT0000 complete [ 1201.632555] LustreError: 35797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.651047] LustreError: 35797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1204.793064] LustreError: 39130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748640 with bad export cookie 4191780556524382866 [ 1204.805681] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.814280] LustreError: 39130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1205.360853] Lustre: server umount lustre-MDT0001 complete [ 1218.447123] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1222.438144] Key type lgssc unregistered [ 1222.845956] LNet: 42927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.855523] LNetError: 42927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.912387] LNet: Removed LNI 192.168.201.110@tcp [ 1224.745762] Key type .llcrypt unregistered [ 1224.747685] Key type ._llcrypt unregistered [ 1239.378356] Key type ._llcrypt registered [ 1239.381872] Key type .llcrypt registered [ 1239.492968] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1253.032589] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1254.465305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.486119] alg: No test for adler32 (adler32-zlib) [ 1255.565237] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1255.814441] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1257.471283] Key type lgssc registered [ 1258.338466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.511815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.527253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.752773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1289.781964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1289.855775] Lustre: lustre-MDT0000: new disk, initializing [ 1289.918084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.939817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1294.820933] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1305.910440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.016595] Lustre: 46077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1306.062636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1306.066697] Lustre: Skipped 1 previous similar message [ 1306.133282] Lustre: lustre-MDT0001: new disk, initializing [ 1306.224520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.264221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1306.274564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1310.169490] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1319.842303] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.187886] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1337.380348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.785336] Lustre: lustre-OST0000: new disk, initializing [ 1337.796854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1337.810757] Lustre: 48049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1337.924126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.421895] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1344.600121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1344.617125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1344.768324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1354.272719] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1359.168196] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1359.521968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1364.882774] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1365.333630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1370.595547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.604320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1370.608751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.609410] LustreError: Skipped 1 previous similar message [ 1370.625436] Lustre: Skipped 1 previous similar message [ 1373.650558] Lustre: server umount lustre-OST0000 complete [ 1377.760682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1377.767085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.782584] Lustre: Skipped 1 previous similar message [ 1377.795092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.833802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.835364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.983601] Lustre: server umount lustre-MDT0000 complete [ 1387.568523] LustreError: 46883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748823 with bad export cookie 6542103638518469844 [ 1387.572559] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.578759] LustreError: 46883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1387.843925] Lustre: server umount lustre-MDT0001 complete [ 1396.611602] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:07:11 (1786748831) [ 1403.628744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.235459] LustreError: 50508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.253706] LustreError: 50508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1404.313336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.885912] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1409.511132] LustreError: 50509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.630751] LustreError: 50508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.654707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.176345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.835323] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1426.424702] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1432.029916] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1438.317091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.825268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.699448] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1454.442305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.335279] LustreError: 53038:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1460.516302] Lustre: Modifying parameter general.timeout=1 in log params [ 1461.583295] Lustre: Modifying parameter general.timeout=2 in log params [ 1464.656447] Lustre: Modifying parameter general.timeout=5 in log params [ 1464.660587] Lustre: Skipped 2 previous similar messages [ 1469.077688] Lustre: Modifying parameter general.timeout=10 in log params [ 1469.080816] Lustre: Skipped 4 previous similar messages [ 1477.338227] Lustre: Modifying parameter general.timeout=20 in log params [ 1477.342423] Lustre: Skipped 9 previous similar messages [ 1493.790376] Lustre: Modifying parameter general.timeout=43 in log params [ 1493.798874] Lustre: Skipped 22 previous similar messages [ 1526.196878] Lustre: Modifying parameter general.timeout=82 in log params [ 1526.200656] Lustre: Skipped 38 previous similar messages [ 1580.759321] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:10:15 (1786749015) [ 1583.419616] Lustre: 59982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1585.146855] Lustre: 60080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1587.284760] Lustre: 60194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1591.709799] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1591.715984] Lustre: Skipped 54 previous similar messages [ 1592.466437] Lustre: 60499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1592.483548] Lustre: 60499:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1617.114446] Lustre: 61857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1617.147108] LustreError: 61857:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1622.161426] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:10:56 (1786749056) [ 1636.336521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.357476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.369688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.380864] Lustre: Skipped 1 previous similar message [ 1641.288455] Lustre: server umount lustre-MDT0000 complete [ 1644.516159] LustreError: 50509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.527880] LustreError: 50509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1645.111022] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749081 with bad export cookie 6542103638518470726 [ 1645.113952] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.118100] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1645.415974] Lustre: server umount lustre-MDT0001 complete [ 1659.929922] Lustre: server umount lustre-OST0000 complete [ 1690.243895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.608257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1691.039376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.400160] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1706.123176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.255298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1706.295794] Lustre: 64390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1710.533958] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1719.318985] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.116099] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1730.617434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.865195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1731.137560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.149519] Lustre: Skipped 1 previous similar message [ 1736.939609] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1744.441453] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.798898] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1745.813978] Lustre: Skipped 16 previous similar messages [ 1745.863489] Lustre: 66956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1752.032740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1752.039831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.069882] Lustre: Skipped 2 previous similar messages [ 1752.075225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.092182] Lustre: Skipped 3 previous similar messages [ 1757.978268] Lustre: server umount lustre-OST0000 complete [ 1762.791453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.792506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1762.793058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.793063] Lustre: Skipped 3 previous similar messages [ 1762.817612] Lustre: Skipped 2 previous similar messages [ 1762.819473] LustreError: Skipped 1 previous similar message [ 1768.345976] Lustre: server umount lustre-MDT0000 complete [ 1772.159932] LustreError: 66252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749208 with bad export cookie 6542103638518473694 [ 1772.160549] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.178467] LustreError: 66252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1772.647472] Lustre: server umount lustre-MDT0001 complete [ 1782.907150] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1785.191645] Key type lgssc unregistered [ 1785.460594] LNet: 68165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.468187] LNetError: 68165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.534594] LNet: Removed LNI 192.168.201.110@tcp [ 1787.134519] Key type .llcrypt unregistered [ 1787.138418] Key type ._llcrypt unregistered [ 1814.329859] Key type ._llcrypt registered [ 1814.331854] Key type .llcrypt registered [ 1814.495848] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1816.686673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.807739] alg: No test for adler32 (adler32-zlib) [ 1818.097567] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1818.508298] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1820.299512] Key type lgssc registered [ 1821.183650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.839035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.897615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.728105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.968532] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1841.121957] LustreError: 69579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.156876] LustreError: 69579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1844.945269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.967373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1850.363668] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1858.519545] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1865.631276] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1873.055711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.053633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.287172] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1890.306622] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1898.307844] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1907.173249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.195041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.235642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.270213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.275523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.298017] Lustre: Skipped 1 previous similar message [ 1911.309939] Lustre: Skipped 2 previous similar messages [ 1912.963542] Lustre: server umount lustre-MDT0000 complete [ 1916.384711] LustreError: 69574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.412789] LustreError: 69574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1917.170799] LustreError: 70318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749353 with bad export cookie 5430621508425591100 [ 1917.182371] LustreError: 70318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1917.183418] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.385770] Lustre: server umount lustre-MDT0001 complete [ 1931.983227] Lustre: server umount lustre-OST0000 complete [ 1960.686178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.971933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1961.004812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1961.292104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.021302] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1976.729445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.793202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1976.815453] Lustre: 74453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.205209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.214888] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1989.731440] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1995.501167] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2001.630428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2001.950301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2002.442737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.803791] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2018.556499] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2020.286920] Lustre: 77030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2022.388806] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2029.024791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2029.037287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.059961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.883442] Lustre: server umount lustre-OST0000 complete [ 2039.778552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.780786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.813977] Lustre: Skipped 2 previous similar messages [ 2039.831609] Lustre: Skipped 2 previous similar messages [ 2044.900927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.912163] Lustre: Skipped 1 previous similar message [ 2053.087169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2053.381338] Lustre: server umount lustre-MDT0000 complete [ 2055.141586] LustreError: 75372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.175858] LustreError: 75372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2057.424279] LustreError: 74447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749493 with bad export cookie 5430621508425591947 [ 2057.428197] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2057.440857] LustreError: 74447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2058.150741] Lustre: server umount lustre-MDT0001 complete [ 2069.219390] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2072.130503] Key type lgssc unregistered [ 2072.387648] LNet: 78245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.394474] LNetError: 78245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.405843] LNet: Removed LNI 192.168.201.110@tcp [ 2073.517860] Key type .llcrypt unregistered [ 2073.523618] Key type ._llcrypt unregistered [ 2101.087311] Key type ._llcrypt registered [ 2101.088965] Key type .llcrypt registered [ 2101.181438] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2102.410919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.487343] alg: No test for adler32 (adler32-zlib) [ 2103.618229] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2104.082073] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2106.017434] Key type lgssc registered [ 2107.497988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2118.544740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2118.594169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.469557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.185068] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2130.915701] LustreError: 79658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.924057] LustreError: 79658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2134.466527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.398090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.646467] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2147.866917] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.359728] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.924672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.055880] LustreError: 81485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.621438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.076109] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2175.018099] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.857989] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.749712] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2184.368137] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2184.378446] Lustre: Skipped 1 previous similar message [ 2186.816693] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2186.823874] Lustre: Skipped 2 previous similar messages [ 2191.227148] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2191.230739] Lustre: Skipped 4 previous similar messages [ 2200.060273] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2200.075198] Lustre: Skipped 9 previous similar messages [ 2216.630830] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2216.639460] Lustre: Skipped 17 previous similar messages [ 2249.057445] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2249.060965] Lustre: Skipped 35 previous similar messages [ 2278.318528] Lustre: 87758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2280.831202] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:21:55 (1786749715) [ 2282.813575] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2284.762233] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:21:59 (1786749719) [ 2293.728027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2293.734719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.738920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2293.756329] Lustre: Skipped 1 previous similar message [ 2296.051742] Lustre: server umount lustre-OST0000 complete [ 2303.968816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.981613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2303.986988] LustreError: Skipped 1 previous similar message [ 2303.990430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.005122] Lustre: Skipped 2 previous similar messages [ 2307.990992] Lustre: server umount lustre-MDT0000 complete [ 2309.089058] LustreError: 79654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.123204] LustreError: 79654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2312.268842] LustreError: 81510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749748 with bad export cookie 5555268846768413597 [ 2312.271718] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.278938] LustreError: 81510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2313.004274] Lustre: server umount lustre-MDT0001 complete [ 2324.687881] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2327.719370] Key type lgssc unregistered [ 2328.082553] LNet: 89010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.097580] LNetError: 89010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.125896] LNet: Removed LNI 192.168.201.110@tcp [ 2330.013750] Key type .llcrypt unregistered [ 2330.015653] Key type ._llcrypt unregistered [ 2335.172634] Key type ._llcrypt registered [ 2335.179503] Key type .llcrypt registered [ 2335.344819] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2340.610109] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules [ 2342.121624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.249221] alg: No test for adler32 (adler32-zlib) [ 2343.277657] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2343.516984] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2345.231213] Key type lgssc registered [ 2346.860057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2348.362890] LustreError: 90227:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2363.949451] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2369.561190] LustreError: 90227:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2375.073345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.084583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.736769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.426974] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2386.913583] LustreError: 91237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.930924] LustreError: 91237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2390.012995] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:23:44 (1786749824) [ 2392.038476] LustreError: 91236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.024282] LustreError: 91269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2396.052408] LustreError: 91269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2396.064554] LustreError: 91269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2396.568897] Lustre: server umount lustre-MDT0000 complete [ 2408.585776] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2411.132397] Key type lgssc unregistered [ 2411.524531] LNet: 92767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.528292] LNetError: 92767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.586902] LNet: Removed LNI 192.168.201.110@tcp [ 2412.297388] Key type .llcrypt unregistered [ 2412.303734] Key type ._llcrypt unregistered [ 2434.216795] Key type ._llcrypt registered [ 2434.219506] Key type .llcrypt registered [ 2434.369254] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2435.704146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2435.805590] alg: No test for adler32 (adler32-zlib) [ 2436.983396] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2437.360395] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2439.135627] Key type lgssc registered [ 2440.439894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.328720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2448.366296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.071472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.499058] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2459.618635] LustreError: 93988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.640623] LustreError: 93988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2463.016649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.825250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.283893] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2478.184430] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2485.429695] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2492.056829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.595973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.629186] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2506.169890] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2555.810162] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:26:30 (1786749990) [ 2557.325514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2559.294225] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:26:33 (1786749993) [ 2565.601738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.608069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.624990] Lustre: Skipped 2 previous similar messages [ 2565.631893] Lustre: Skipped 2 previous similar messages [ 2570.721850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.741222] Lustre: Skipped 2 previous similar messages [ 2571.334612] Lustre: server umount lustre-MDT0000 complete [ 2575.092828] LustreError: 93968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750011 with bad export cookie 14537114597121243899 [ 2575.093858] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.105671] LustreError: 93968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2575.487068] Lustre: server umount lustre-MDT0001 complete [ 2590.282049] Lustre: server umount lustre-OST0000 complete [ 2601.788295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.153704] LustreError: 98265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.326463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.740854] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2607.594923] LustreError: 98266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.705419] LustreError: 98265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.487524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.346474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2620.637819] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2628.073668] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2633.741992] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2648.247342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.601319] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2648.606870] LustreError: 98253:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2648.630097] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2648.640480] LustreError: 100349:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2648.647203] LustreError: 100349:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2648.652179] LustreError: 100349:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2648.784265] Lustre: server umount lustre-OST0000 complete [ 2648.794623] LustreError: 100349:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2658.918493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.076360] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2659.087406] LustreError: 98255:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2659.131344] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2659.145353] LustreError: 100816:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2659.154405] LustreError: 100816:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2659.162732] LustreError: 100816:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2659.236876] Lustre: server umount lustre-OST0000 complete [ 2659.243481] LustreError: 100816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2670.242597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.527501] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2670.534116] Lustre: Found index 0 for lustre-OST0000, updating log [ 2670.544804] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2670.698865] Lustre: lustre-OST0000: new disk, initializing [ 2670.709793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2670.724190] Lustre: 101351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2671.090295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.444501] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2683.608215] Lustre: Failing over lustre-OST0000 [ 2683.764383] Lustre: server umount lustre-OST0000 complete [ 2691.047251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.058306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.068370] Lustre: Skipped 1 previous similar message [ 2691.108946] Lustre: Skipped 1 previous similar message [ 2696.160635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.170495] Lustre: Skipped 1 previous similar message [ 2701.281578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.295079] Lustre: Skipped 1 previous similar message [ 2702.305874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2702.507521] Lustre: server umount lustre-MDT0000 complete [ 2706.394949] LustreError: 98246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750142 with bad export cookie 14537114597121246643 [ 2706.405742] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.409550] LustreError: 98246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2706.713332] Lustre: server umount lustre-MDT0001 complete [ 2716.191159] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:29:10 (1786750150) [ 2732.762841] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2742.066483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.348883] LustreError: 103697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.442395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.761877] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2752.992391] LustreError: 103698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.008642] LustreError: 103698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2753.935886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.832464] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2762.496863] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2768.759137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.320868] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2776.104807] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2779.558648] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2779.563746] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2779.572954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2784.740893] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2784.774851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2784.853626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.008879] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2785.232621] Lustre: lustre-OST0001: new disk, initializing [ 2785.239286] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2785.247730] Lustre: 106014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2785.704891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.707879] Lustre: Skipped 2 previous similar messages [ 2787.764531] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:01 [ 2787.794838] Lustre: Skipped 1 previous similar message [ 2787.814190] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2787.823151] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2787.842632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2787.915995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2787.939541] LustreError: Skipped 1 previous similar message [ 2794.044279] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2795.164680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2795.187161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2795.313402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2801.520312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2807.712742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2812.304660] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 2815.974183] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.978930] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.982093] Lustre: Skipped 1 previous similar message [ 2817.918146] Lustre: server umount lustre-MDT0001 complete [ 2821.101660] LustreError: 103698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.122525] LustreError: 103698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2827.480939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.551141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2827.558600] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2827.603226] Lustre: 103687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2829.183766] LustreError: 107596:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2829.188771] LustreError: 107596:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2829.194700] LustreError: 103690:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2829.308864] LustreError: 107601:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 2829.315932] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2829.315932] [ 2832.587114] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2833.394737] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2842.444988] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:31:17 (1786750277) [ 2859.980603] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2877.189869] Lustre: 109631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2903.106861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2906.551763] Lustre: 110767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3000.291424] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.312904] Lustre: Skipped 1 previous similar message [ 3000.320733] LustreError: 106954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.351957] LustreError: 106954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3000.690904] Lustre: server umount lustre-MDT0000 complete [ 3005.920319] LustreError: 103680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750441 with bad export cookie 14537114597121247644 [ 3005.930732] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.959236] LustreError: 103680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3006.498844] Lustre: server umount lustre-MDT0001 complete [ 3022.575500] Lustre: server umount lustre-OST0000 complete [ 3036.364065] Lustre: server umount lustre-OST0001 complete [ 3043.958554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3057.154390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3067.235408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3080.354659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3105.529455] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3115.680783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.696636] Lustre: lustre-MDT0000: reset Object Index mappings [ 3116.028967] LustreError: 114725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.042900] LustreError: 114725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3116.128088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.134046] Lustre: Skipped 1 previous similar message [ 3120.838303] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3130.023948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.044710] Lustre: lustre-MDT0001: reset Object Index mappings [ 3130.336702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3130.345047] Lustre: Skipped 4 previous similar messages [ 3134.887739] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3137.810725] Lustre: 115873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3145.622562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.614197] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3155.277964] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3159.891546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.382220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3166.229154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3166.237650] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3167.491990] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3174.863513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3178.412192] Lustre: 117748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3223.668419] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:37:37 (1786750657) [ 3253.226636] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.237752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.251697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.256758] Lustre: Skipped 6 previous similar messages [ 3253.291781] Lustre: Skipped 3 previous similar messages [ 3256.726862] Lustre: server umount lustre-MDT0000 complete [ 3258.344393] LustreError: 114725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.372484] LustreError: 114725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3260.241222] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750696 with bad export cookie 14537114597121294740 [ 3260.267592] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.268294] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3263.459250] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.484086] Lustre: Skipped 1 previous similar message [ 3267.000790] Lustre: server umount lustre-MDT0001 complete [ 3277.330848] Lustre: server umount lustre-OST0000 complete [ 3288.022796] Lustre: server umount lustre-OST0001 complete [ 3296.513981] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3309.220523] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3351.616962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.821289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3351.863876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3351.965568] Lustre: lustre-MDT0000: new disk, initializing [ 3352.058350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.074633] Lustre: Skipped 3 previous similar messages [ 3352.092369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3357.162718] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3366.878925] Lustre: server umount lustre-MDT0000 complete [ 3377.441678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.753840] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3377.783838] Lustre: 122018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.803483] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3383.190496] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3388.869500] Lustre: server umount lustre-MDT0000 complete [ 3397.239259] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:40:32 (1786750832) [ 3398.968857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3400.866856] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:40:35 (1786750835) [ 3418.796680] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3429.221242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3434.441527] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3445.236546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.390994] Lustre: 124065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3445.444260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3445.447568] Lustre: Skipped 1 previous similar message [ 3445.522429] Lustre: lustre-MDT0001: new disk, initializing [ 3445.662777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3445.680576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3450.764510] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3455.925509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3463.120236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.333583] Lustre: lustre-OST0000: new disk, initializing [ 3463.336891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3463.341934] Lustre: 125647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3464.615223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3464.627576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3464.759325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3469.860930] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3480.176720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.387354] Lustre: lustre-OST0001: new disk, initializing [ 3480.391477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3480.396381] Lustre: 126518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3481.982418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3481.995508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3482.116320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3488.006818] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3497.738311] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3501.778894] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3907.290790] Lustre: 132933:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3907.300181] Lustre: 132933:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3907.309830] Lustre: 132933:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3907.319756] Lustre: 132933:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3907.334288] Lustre: 132933:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3907.342356] Lustre: 132933:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4168.198811] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:53:21 (1786751601) [ 4184.544925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.546395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4184.575437] Lustre: Skipped 3 previous similar messages [ 4184.591841] Lustre: Skipped 7 previous similar messages [ 4190.739547] Lustre: server umount lustre-MDT0000 complete [ 4194.242739] LustreError: 124059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751630 with bad export cookie 14537114597121359714 [ 4194.248050] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.250446] LustreError: 124059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4194.784256] LustreError: 124072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.790325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.798374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4194.798382] Lustre: Skipped 4 previous similar messages [ 4194.820434] LustreError: 124072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4194.873813] Lustre: Skipped 1 previous similar message [ 4201.202848] Lustre: server umount lustre-MDT0001 complete [ 4211.732459] Lustre: server umount lustre-OST0000 complete [ 4230.111136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4230.541461] Lustre: server umount lustre-OST0001 complete [ 4239.295721] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4252.258194] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4293.432344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.817730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4293.867704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4293.944865] Lustre: lustre-MDT0000: new disk, initializing [ 4294.079988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.089571] Lustre: Skipped 5 previous similar messages [ 4294.107185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4300.556519] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4311.905947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.076828] Lustre: 135984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4312.130858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4312.137540] Lustre: Skipped 1 previous similar message [ 4312.202829] Lustre: lustre-MDT0001: new disk, initializing [ 4312.320157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4312.332324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4317.073661] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4327.056110] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4332.550126] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4339.102712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4339.359157] Lustre: lustre-OST0000: new disk, initializing [ 4339.363505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4339.367841] Lustre: 137952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4339.453041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.460261] Lustre: Skipped 1 previous similar message [ 4340.590381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4340.608926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4340.729889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4346.336793] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4355.542601] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.574972] Lustre: lustre-MDD0000: changelog on [ 4368.592694] Lustre: lustre-MDD0001: changelog on [ 5976.532929] Lustre: lustre-MDD0001: changelog off [ 5978.396292] Lustre: 137650:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5979.209781] Lustre: lustre-MDD0000: changelog off [ 5988.834197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.839895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.853544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.863137] Lustre: Skipped 3 previous similar messages [ 5989.350076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.467874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.478149] Lustre: Skipped 3 previous similar messages [ 5999.071155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5999.240540] Lustre: server umount lustre-OST0000 complete [ 6004.707111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.708415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.727048] Lustre: Skipped 1 previous similar message [ 6004.737499] Lustre: Skipped 1 previous similar message [ 6008.549042] Lustre: server umount lustre-MDT0000 complete [ 6009.835063] LustreError: 135991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.870888] LustreError: 135991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6012.231779] LustreError: 137951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753448 with bad export cookie 14537114597122139710 [ 6012.234053] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.239147] LustreError: 137951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6012.668485] Lustre: server umount lustre-MDT0001 complete [ 6022.712841] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 6025.280567] Key type lgssc unregistered [ 6025.609773] LNet: 147064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.618035] LNetError: 147064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6025.633950] LNet: Removed LNI 192.168.201.110@tcp [ 6026.366168] Key type .llcrypt unregistered [ 6026.370612] Key type ._llcrypt unregistered [ 6045.118210] Key type ._llcrypt registered [ 6045.123858] Key type .llcrypt registered [ 6045.228972] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6058.832135] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6059.853501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6059.931697] alg: No test for adler32 (adler32-zlib) [ 6060.982679] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 6061.232353] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 6062.871169] Key type lgssc registered [ 6063.822800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.240959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6099.268743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.466197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6100.528198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6100.634518] Lustre: lustre-MDT0000: new disk, initializing [ 6100.725136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.749970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6104.626598] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6114.984582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.115993] Lustre: 150283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6115.158797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6115.162966] Lustre: Skipped 1 previous similar message [ 6115.216517] Lustre: lustre-MDT0001: new disk, initializing [ 6115.272113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6115.315426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6115.330618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6119.353941] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6127.939383] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6133.757474] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6140.929124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.163344] Lustre: lustre-OST0000: new disk, initializing [ 6141.168514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6141.175950] Lustre: 152249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6141.336523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.235887] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6149.735301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6149.755755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6149.885672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6157.354370] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.228543] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module obdecho/obdecho [ 6164.970715] LustreError: 153252:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6164.980863] LustreError: 153252:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6166.968285] LustreError: 153348:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6180.837877] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.840091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.860867] Lustre: Skipped 1 previous similar message [ 6180.880877] Lustre: Skipped 2 previous similar messages [ 6185.955608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.960342] Lustre: Skipped 2 previous similar messages [ 6186.863834] Lustre: server umount lustre-MDT0000 complete [ 6190.232271] LustreError: 150274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753626 with bad export cookie 16427602735281091693 [ 6190.243838] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.247839] LustreError: 150274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6190.578320] Lustre: server umount lustre-MDT0001 complete [ 6204.948434] Lustre: server umount lustre-OST0000 complete [ 6225.085516] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6235.912644] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6274.886907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.235476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6275.309342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6275.406637] Lustre: lustre-MDT0000: new disk, initializing [ 6275.524346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.546402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6280.308617] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6292.279384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.378884] Lustre: 157068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6292.460625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6292.462914] Lustre: Skipped 1 previous similar message [ 6292.578467] Lustre: lustre-MDT0001: new disk, initializing [ 6292.749245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6292.831248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6292.839362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6299.887754] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6309.707115] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6316.549535] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6325.951617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.673457] Lustre: lustre-OST0000: new disk, initializing [ 6326.689773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6326.701413] Lustre: 159036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6326.856693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6333.054690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6333.065693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6333.225144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6335.595780] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6346.782415] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6353.520630] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6353.777982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6360.643334] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6360.867040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6364.128876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.141694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.159169] Lustre: Skipped 1 previous similar message [ 6364.167862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.182688] Lustre: Skipped 1 previous similar message [ 6369.078773] Lustre: server umount lustre-OST0000 complete [ 6374.378181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.384586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6374.403953] Lustre: Skipped 2 previous similar messages [ 6374.418036] Lustre: Skipped 1 previous similar message [ 6379.491635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.503662] Lustre: Skipped 1 previous similar message [ 6380.067359] Lustre: server umount lustre-MDT0000 complete [ 6383.744294] LustreError: 157863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753819 with bad export cookie 16427602735281092645 [ 6383.752834] LustreError: 157863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6383.760928] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.153702] Lustre: server umount lustre-MDT0001 complete [ 6387.929962] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:30:22 (1786753822) [ 6389.606658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6391.749836] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:30:26 (1786753826) [ 6399.559372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.171806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6404.662622] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6405.626244] LustreError: 161449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.653297] LustreError: 161449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6410.720585] LustreError: 161450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.881328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.509458] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6425.469146] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6431.053434] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.297785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.569581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6437.576824] Lustre: Skipped 1 previous similar message [ 6443.030844] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6451.699606] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6457.747201] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [ 6460.386092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6460.397207] LustreError: Skipped 1 previous similar message [ 6460.399672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.209278] Lustre: server umount lustre-MDT0000 complete [ 6466.529306] LustreError: 161450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6466.557532] LustreError: 161450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6466.943349] LustreError: 161430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753902 with bad export cookie 16427602735281093534 [ 6466.952298] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.649137] LustreError: 161444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.651425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.668166] LustreError: 161444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6471.698857] Lustre: Skipped 2 previous similar messages [ 6476.768704] LustreError: 162730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.769055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6476.805081] LustreError: 162730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6476.850059] Lustre: Skipped 8 previous similar messages [ 6481.375185] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6481.753443] Lustre: server umount lustre-MDT0001 complete [ 6486.846175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6499.814253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.113922] LustreError: 165290:0:(llog.c:1655:llog_backup()) MGC192.168.201.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6500.125566] Lustre: 165290:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6507.492195] LustreError: 165295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.509913] LustreError: 165295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6525.323567] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.343736] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6558.177793] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.202779] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 6560.736634] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6560.741863] Lustre: 165290:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6601.723150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.640562] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6606.828279] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6615.998464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.202115] LustreError: 165297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.227823] LustreError: 165297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 46 previous similar messages [ 6628.325635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6637.535927] LustreError: 148505:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002c7be4d9 x1873546835005952/t0(0) o250->MGC192.168.201.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6643.059656] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6643.205665] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6651.037667] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6652.582588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6658.859766] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.310865] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6679.014661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.019079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.027861] Lustre: Skipped 2 previous similar messages [ 6679.041503] Lustre: Skipped 4 previous similar messages [ 6684.731294] Lustre: server umount lustre-MDT0000 complete [ 6688.453510] LustreError: 161431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754124 with bad export cookie 16427602735281100023 [ 6688.461653] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.473568] LustreError: 161431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6689.107259] Lustre: server umount lustre-MDT0001 complete [ 6702.435793] Lustre: server umount lustre-OST0000 complete [ 6726.603873] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6738.602969] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6781.094357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.447918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6781.480395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6781.578700] Lustre: lustre-MDT0000: new disk, initializing [ 6781.733674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.747274] Lustre: Skipped 1 previous similar message [ 6781.771642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6786.993555] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6798.838552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6798.978778] Lustre: 170399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6799.228941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6804.636948] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6814.963789] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6822.260338] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6829.945563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.296882] Lustre: lustre-OST0000: new disk, initializing [ 6830.301228] Lustre: Skipped 1 previous similar message [ 6830.306361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6830.316293] Lustre: Skipped 2 previous similar messages [ 6830.331920] Lustre: 172367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6837.268212] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6839.354823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6839.365946] Lustre: Skipped 1 previous similar message [ 6839.375473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6839.467888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6848.168842] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6853.629500] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6853.874559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6860.088378] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6860.402944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6865.382330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6865.397516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.398725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6865.413314] Lustre: Skipped 1 previous similar message [ 6865.428618] Lustre: Skipped 4 previous similar messages [ 6868.846488] Lustre: server umount lustre-OST0000 complete [ 6879.067034] Lustre: server umount lustre-MDT0000 complete [ 6880.737212] LustreError: 170410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6880.764488] LustreError: 170410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6883.245914] LustreError: 170391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754319 with bad export cookie 16427602735281116823 [ 6883.258852] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6883.260467] LustreError: 170391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6883.613508] Lustre: server umount lustre-MDT0001 complete [ 6892.067424] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:38:46 (1786754326) [ 6898.384771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.723956] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6912.570714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.680259] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6927.230250] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.860455] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6942.207343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.741554] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6957.917723] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.628549] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:40:02 (1786754402) [ 6972.896313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.902299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.907741] LustreError: Skipped 1 previous similar message [ 6972.932749] Lustre: Skipped 3 previous similar messages [ 6977.925596] Lustre: server umount lustre-OST0000 complete [ 6987.640162] Lustre: server umount lustre-MDT0000 complete [ 6991.161152] LustreError: 174761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754427 with bad export cookie 16427602735281117705 [ 6991.170119] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.177679] LustreError: 174761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6991.460628] Lustre: server umount lustre-MDT0001 complete [ 7002.061799] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 7005.019924] Key type lgssc unregistered [ 7005.244796] LNet: 178509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7005.252307] LNetError: 178509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7005.265704] LNet: Removed LNI 192.168.201.110@tcp [ 7006.126569] Key type .llcrypt unregistered [ 7006.128190] Key type ._llcrypt unregistered [ 7024.681722] Key type ._llcrypt registered [ 7024.683824] Key type .llcrypt registered [ 7024.815826] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7026.177384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7026.230403] alg: No test for adler32 (adler32-zlib) [ 7027.383752] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 7027.682397] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 7029.343249] Key type lgssc registered [ 7030.221966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7037.971966] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7037.996457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.599922] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7058.719455] LustreError: 179755:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 7063.841852] LustreError: 179755:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7063.856547] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7063.861206] LustreError: MGC192.168.201.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7063.875020] LustreError: 179755:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7063.888711] LustreError: 179755:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7063.903563] LustreError: 179755:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7064.006488] Lustre: server umount lustre-OST0000 complete [ 7064.011554] LustreError: 179755:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7073.374663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.876027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.807937] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7085.046672] LustreError: 180255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.097557] LustreError: 180255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7089.815700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.157425] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7090.349087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7095.625680] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7103.822104] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7110.105504] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7117.113665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7117.251734] LustreError: 182071:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7117.266514] Lustre: 182071:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7117.392362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.845336] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7131.746571] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.203872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.208568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.224027] Lustre: Skipped 2 previous similar messages [ 7135.243186] Lustre: Skipped 2 previous similar messages [ 7140.351287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7140.364466] Lustre: Skipped 2 previous similar messages [ 7140.628958] Lustre: server umount lustre-MDT0000 complete [ 7144.281818] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754580 with bad export cookie 1881882312956904464 [ 7144.298215] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.303664] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7145.444572] LustreError: 180250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.454748] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.475856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7145.503365] LustreError: 180983:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005eb75230 x1873547847823360/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7145.661843] Lustre: server umount lustre-MDT0001 complete [ 7151.922188] Lustre: server umount lustre-OST0000 complete [ 7161.409262] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7173.164789] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7210.542418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.048624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7211.112727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7211.248952] Lustre: lustre-MDT0000: new disk, initializing [ 7211.547853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.584237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7217.963664] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7231.173104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.316231] Lustre: 185597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7231.379169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7231.390444] Lustre: Skipped 1 previous similar message [ 7231.466392] Lustre: lustre-MDT0001: new disk, initializing [ 7231.557268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.593444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7231.604541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7236.361356] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7245.797240] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7252.308876] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7261.825305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.126288] Lustre: lustre-OST0000: new disk, initializing [ 7262.129154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7262.139795] Lustre: 187565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7262.309636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7272.122173] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7272.535130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7272.546745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7272.699516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7282.108759] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7286.746344] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7286.924638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7290.866509] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7291.101200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7293.409513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7293.421088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7293.422213] LustreError: Skipped 1 previous similar message [ 7293.435435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7293.447508] Lustre: Skipped 1 previous similar message [ 7299.296790] Lustre: server umount lustre-OST0000 complete [ 7303.136690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7303.143498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.157562] Lustre: Skipped 1 previous similar message [ 7303.163606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.169237] Lustre: Skipped 2 previous similar messages [ 7309.385854] Lustre: server umount lustre-MDT0000 complete [ 7312.994028] LustreError: 187564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754748 with bad export cookie 1881882312956905185 [ 7313.004724] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.011457] LustreError: 187564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7313.640950] Lustre: server umount lustre-MDT0001 complete [ 7322.713693] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:45:57 (1786754757) [ 7338.568438] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 7342.182114] Key type lgssc unregistered [ 7342.560863] LNet: 190513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7342.590394] LNetError: 190513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7342.627364] LNet: Removed LNI 192.168.201.110@tcp [ 7343.678924] Key type .llcrypt unregistered [ 7343.687996] Key type ._llcrypt unregistered [ 7367.093980] Key type ._llcrypt registered [ 7367.099174] Key type .llcrypt registered [ 7367.227679] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7368.981969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7369.082099] alg: No test for adler32 (adler32-zlib) [ 7370.233503] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 7370.497496] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 7372.127221] Key type lgssc registered [ 7373.278694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7381.941535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7381.949226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.347788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.359526] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7393.771518] LustreError: 191732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.802420] LustreError: 191732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7395.534994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.015484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.991957] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7407.024271] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7413.337708] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7421.208155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7421.544233] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7421.549204] LustreError: 193598:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7421.564054] Lustre: 193598:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7421.715018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.709495] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7437.351722] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.006991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.013780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.033176] Lustre: Skipped 2 previous similar messages [ 7441.055313] Lustre: Skipped 2 previous similar messages [ 7444.072058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.086222] Lustre: Skipped 2 previous similar messages [ 7446.275436] Lustre: server umount lustre-MDT0000 complete [ 7449.578312] LustreError: 191729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.603763] LustreError: 191729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7450.485381] LustreError: 191712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754886 with bad export cookie 13550793552215177455 [ 7450.498882] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7451.359631] LustreError: 192464:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000358390fb x1873548207486208/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7451.517196] Lustre: server umount lustre-MDT0001 complete [ 7466.208494] Lustre: server umount lustre-OST0000 complete [ 7477.448417] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7488.802792] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7528.372610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.871926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7528.946380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7529.163845] Lustre: lustre-MDT0000: new disk, initializing [ 7529.396724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.441681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7535.632373] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7549.720703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.869719] Lustre: 197124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7549.965342] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7549.967906] Lustre: Skipped 1 previous similar message [ 7549.971365] LustreError: 197907:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7549.978960] LustreError: 197907:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7549.983321] Lustre: 197907:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7550.000397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7550.004079] Lustre: Skipped 1 previous similar message [ 7550.057615] Lustre: lustre-MDT0001: new disk, initializing [ 7550.176461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7550.222330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7550.243462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7554.049312] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7564.894408] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.204937] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7578.219191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.534650] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7578.538585] Lustre: Skipped 1 previous similar message [ 7578.542103] LustreError: 199071:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7578.555969] LustreError: 199071:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7578.563463] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7578.583446] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7578.608000] Lustre: lustre-OST0000: new disk, initializing [ 7578.611429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7578.615966] Lustre: 199095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7578.709826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.081421] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7588.449677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7588.464716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7588.600578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7595.470780] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7600.207173] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7600.519895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7605.418669] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7605.656544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7609.322462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7609.327903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.336576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.348988] Lustre: Skipped 1 previous similar message [ 7613.894823] Lustre: server umount lustre-OST0000 complete [ 7619.553629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.560072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7619.565596] Lustre: Skipped 1 previous similar message [ 7619.578632] Lustre: Skipped 2 previous similar messages [ 7624.673553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.682087] Lustre: Skipped 1 previous similar message [ 7632.351596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7632.575483] Lustre: server umount lustre-MDT0000 complete [ 7634.914750] LustreError: 197136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.940862] LustreError: 197136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7636.866554] LustreError: 197116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755072 with bad export cookie 13550793552215178176 [ 7636.882076] LustreError: 197116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7636.889869] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7637.336032] Lustre: server umount lustre-MDT0001 complete [ 7648.204443] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:51:22 (1786755082) [ 7666.192562] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7676.309904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.711937] LustreError: 202223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.753453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.640600] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7682.018139] LustreError: 202224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.758180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.873264] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7696.736499] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7703.437313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.523081] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7717.979980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.188248] Lustre: lustre-OST0001: new disk, initializing [ 7718.191761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7718.197608] Lustre: 204529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7718.306523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.312526] Lustre: Skipped 2 previous similar messages [ 7724.683978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7724.703550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7724.802514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7725.112716] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7734.918692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7745.372801] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7747.526753] Lustre: DEBUG MARKER: ADD OST3 [ 7752.851994] loop: module loaded [ 7752.985799] loop0: detected capacity change from 0 to 204800000 [ 7753.303650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7753.317553] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7753.336242] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7760.122761] loop0: detected capacity change from 0 to 204800000 [ 7760.449725] Lustre: DEBUG MARKER: STOP OST3 [ 7766.645549] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.930043] Lustre: lustre-OST0003: new disk, initializing [ 7766.938518] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7766.949780] Lustre: 206574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7768.470112] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7768.690161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7768.719598] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7768.803306] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7770.194081] Lustre: Failing over lustre-OST0003 [ 7770.360622] Lustre: server umount lustre-OST0003 complete [ 7770.527145] LustreError: 206694:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7773.582366] LustreError: 204856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.596061] LustreError: 204856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7773.672608] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7773.677035] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.685888] Lustre: Skipped 1 previous similar message [ 7773.973850] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7782.064308] Lustre: DEBUG MARKER: START OST3 again [ 7788.812665] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7789.121297] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7789.130750] Lustre: Skipped 1 previous similar message [ 7789.147504] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7790.898171] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7791.080208] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7791.081645] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7795.387512] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7816.603035] Lustre: Failing over lustre-OST0003 [ 7816.683890] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7816.686889] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.696692] Lustre: Skipped 1 previous similar message [ 7816.701966] LustreError: 203708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.714135] LustreError: 203708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7816.731871] Lustre: server umount lustre-OST0003 complete [ 7835.114474] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.127306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.132749] Lustre: Skipped 3 previous similar messages [ 7841.568425] Lustre: server umount lustre-MDT0000 complete [ 7845.428703] LustreError: 202203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755281 with bad export cookie 13550793552215179058 [ 7845.430480] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.448544] LustreError: 202203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7850.467614] LustreError: 202220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7850.490556] LustreError: 202220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 7852.001944] Lustre: server umount lustre-MDT0001 complete [ 7855.396630] Lustre: server umount lustre-OST0000 complete [ 7859.594940] Lustre: server umount lustre-OST0001 complete [ 7866.687156] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7874.853860] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7912.643169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7912.960858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7913.014128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7913.133056] Lustre: lustre-MDT0000: new disk, initializing [ 7913.271542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7918.706455] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7929.858350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.970899] Lustre: 211988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7930.037117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7930.048601] Lustre: Skipped 1 previous similar message [ 7930.111659] Lustre: lustre-MDT0001: new disk, initializing [ 7930.191398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.197646] Lustre: Skipped 1 previous similar message [ 7930.223186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7930.239715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7934.665322] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7944.566824] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7950.633143] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7958.424221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7958.793095] Lustre: 213958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7960.109205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7960.250837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7964.677545] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7974.618603] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.537437] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7979.797835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7985.427702] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7985.802434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7990.753386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7990.761101] LustreError: Skipped 1 previous similar message [ 7990.764223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.770697] Lustre: Skipped 5 previous similar messages [ 7990.774862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7990.778943] Lustre: Skipped 8 previous similar messages [ 7994.469933] Lustre: server umount lustre-OST0000 complete [ 8004.906844] Lustre: server umount lustre-MDT0000 complete [ 8006.626621] LustreError: 211999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8009.668364] LustreError: 212791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755445 with bad export cookie 13550793552215181354 [ 8009.673055] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8009.679930] LustreError: 212791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8010.366148] Lustre: server umount lustre-MDT0001 complete [ 8014.912613] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:57:29 (1786755449) [ 8033.204796] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8045.625050] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8091.357452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.544820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8091.569353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8091.573369] Lustre: Skipped 1 previous similar message [ 8091.633660] Lustre: lustre-MDT0000: new disk, initializing [ 8091.636876] Lustre: Skipped 1 previous similar message [ 8091.746135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8091.754201] Lustre: Skipped 1 previous similar message [ 8096.800698] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8108.664234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.773199] Lustre: 218570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.078574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8113.264722] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8123.431506] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8130.205958] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8137.459421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.850072] Lustre: 220537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8139.599564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8143.860714] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8154.229096] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8159.849918] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8160.107504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8164.602672] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8164.783912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8170.471552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.474785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8170.486940] Lustre: Skipped 3 previous similar messages [ 8170.490980] LustreError: Skipped 1 previous similar message [ 8170.493842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8170.493849] Lustre: Skipped 4 previous similar messages [ 8172.792330] Lustre: server umount lustre-OST0000 complete [ 8180.707542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8182.895989] Lustre: server umount lustre-MDT0000 complete [ 8185.825538] LustreError: 220616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.862334] LustreError: 220616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8187.117236] LustreError: 220547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755623 with bad export cookie 13550793552215182236 [ 8187.121582] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.129357] LustreError: 220547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8187.584064] Lustre: server umount lustre-MDT0001 complete [ 8196.928600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.566624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.577178] Lustre: Skipped 4 previous similar messages [ 8201.505904] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8210.712522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.124365] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8225.181276] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8231.536767] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8239.996407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8246.631255] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8254.138704] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8267.235362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.237337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8267.252086] Lustre: Skipped 2 previous similar messages [ 8267.258118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8267.281212] Lustre: Skipped 4 previous similar messages [ 8271.195651] Lustre: server umount lustre-OST0000 complete [ 8282.703487] Lustre: server umount lustre-MDT0000 complete [ 8287.000019] LustreError: 222736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755722 with bad export cookie 13550793552215183118 [ 8287.006865] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8287.016629] LustreError: 222736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8287.675845] Lustre: server umount lustre-MDT0001 complete [ 8392.538728] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 8394.905481] Key type lgssc unregistered [ 8395.123873] LNet: 226435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8395.132507] LNetError: 226435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8395.152708] LNet: Removed LNI 192.168.201.110@tcp [ 8395.846515] Key type .llcrypt unregistered [ 8395.848779] Key type ._llcrypt unregistered [ 8421.543714] Key type ._llcrypt registered [ 8421.549966] Key type .llcrypt registered [ 8421.694342] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8435.399293] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8436.320529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8436.449513] alg: No test for adler32 (adler32-zlib) [ 8437.466224] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8437.652225] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 8439.295172] Key type lgssc registered [ 8440.304137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8473.543460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8473.581264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.052522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8475.120453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8475.256033] Lustre: lustre-MDT0000: new disk, initializing [ 8475.420599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8475.439058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8479.987266] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8492.178043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.350399] Lustre: 229787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8492.460095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8492.467701] Lustre: Skipped 1 previous similar message [ 8492.592621] Lustre: lustre-MDT0001: new disk, initializing [ 8492.693121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8492.737715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8492.756908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8497.382924] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8507.103790] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.013541] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.713302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8521.027711] Lustre: lustre-OST0000: new disk, initializing [ 8521.031049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8521.037446] Lustre: 231758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8521.155167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.525190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8527.542840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8527.650052] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8527.717353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8537.338323] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8542.139475] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8542.334185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8547.329546] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8547.522861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8553.453358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.454044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.456726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.465719] Lustre: Skipped 1 previous similar message [ 8555.625823] Lustre: server umount lustre-OST0000 complete [ 8559.584469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8559.595604] LustreError: Skipped 1 previous similar message [ 8559.598443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.608683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.613217] Lustre: Skipped 1 previous similar message [ 8563.684132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8563.687658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8568.802090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8568.808117] Lustre: Skipped 1 previous similar message [ 8573.407132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8573.646722] Lustre: server umount lustre-MDT0000 complete [ 8576.907466] LustreError: 231768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756012 with bad export cookie 9787620337164988568 [ 8576.912501] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.917312] LustreError: 231768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8577.176889] Lustre: server umount lustre-MDT0001 complete [ 8594.575628] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8604.780359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.161271] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8605.194840] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8605.241706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.273377] LustreError: 234694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.807079] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8615.420893] LustreError: 234693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.530147] LustreError: 234694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.704554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.359584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8625.849821] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8629.872077] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8638.597958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.095893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.622590] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8658.403468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8658.658540] Lustre: lustre-OST0001: new disk, initializing [ 8658.662229] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8658.667617] Lustre: 237004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8658.766530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8665.696436] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8667.263255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8667.296983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8667.429206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8676.462715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8682.768973] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8695.261624] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:08:49 (1786756129) [ 8698.348353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.361849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.363475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8698.377642] Lustre: Skipped 3 previous similar messages [ 8698.382079] Lustre: Skipped 2 previous similar messages [ 8703.469546] LustreError: 237451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.498679] LustreError: 237451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8703.606420] Lustre: server umount lustre-MDT0000 complete [ 8707.349547] LustreError: 234675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756143 with bad export cookie 9787620337164989450 [ 8707.357305] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8707.367748] LustreError: 234675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8708.361136] Lustre: server umount lustre-MDT0001 complete [ 8724.620575] Lustre: server umount lustre-OST0000 complete [ 8724.771402] Lustre: 228014:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756144/real 1786756144] req@ffff9bfc7a61c700 x1873549326398976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786756160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8724.800430] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.321309] Lustre: server umount lustre-OST0001 complete [ 8736.230525] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8746.637854] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8785.849315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.105854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8786.130428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8786.191301] Lustre: lustre-MDT0000: new disk, initializing [ 8786.256043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.270298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8790.484721] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8802.546685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8802.659390] Lustre: 240950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8802.701532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8802.704513] Lustre: Skipped 1 previous similar message [ 8802.757736] Lustre: lustre-MDT0001: new disk, initializing [ 8802.849767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8802.861804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8806.569773] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8815.474609] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8821.080506] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8827.413711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8827.695671] Lustre: lustre-OST0000: new disk, initializing [ 8827.698837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8827.703903] Lustre: 242916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8829.747942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8829.754263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8829.875455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8833.539564] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8842.989020] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8847.201312] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8847.424124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8851.822902] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8852.048627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8855.523633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8855.529755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.539187] LustreError: Skipped 1 previous similar message [ 8855.540614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8855.549212] Lustre: Skipped 1 previous similar message [ 8855.581015] Lustre: Skipped 2 previous similar messages [ 8859.805870] Lustre: server umount lustre-OST0000 complete [ 8864.227275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8869.929460] Lustre: server umount lustre-MDT0000 complete [ 8870.882216] LustreError: 240961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.555692] LustreError: 240940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756309 with bad export cookie 9787620337164991592 [ 8873.568773] LustreError: 240940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8873.572085] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8873.913385] Lustre: server umount lustre-MDT0001 complete [ 8881.803252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.293120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8882.297298] Lustre: Skipped 2 previous similar messages [ 8886.733683] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8887.271170] LustreError: 245133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8887.303514] LustreError: 245133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8895.525826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.555384] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8906.646394] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8913.196092] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8919.565372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.953241] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8936.052746] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9079.788627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.799166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.813567] Lustre: Skipped 2 previous similar messages [ 9079.820286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9079.830171] Lustre: Skipped 3 previous similar messages [ 9085.968811] Lustre: server umount lustre-OST0000 complete [ 9104.352884] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9104.573315] Lustre: server umount lustre-MDT0000 complete [ 9106.423581] LustreError: 247575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.448127] LustreError: 247575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9108.789442] LustreError: 245115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756544 with bad export cookie 9787620337164992474 [ 9108.802611] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9109.049770] Lustre: server umount lustre-MDT0001 complete [ 9203.291915] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 9205.322907] Key type lgssc unregistered [ 9205.609651] LNet: 248742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9205.613244] LNetError: 248742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9205.624085] LNet: Removed LNI 192.168.201.110@tcp [ 9206.365601] Key type .llcrypt unregistered [ 9206.367731] Key type ._llcrypt unregistered [ 9233.263767] Key type ._llcrypt registered [ 9233.266598] Key type .llcrypt registered [ 9233.414855] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9252.458995] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9253.635459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9253.704925] alg: No test for adler32 (adler32-zlib) [ 9254.769987] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 9255.128429] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 9256.847277] Key type lgssc registered [ 9258.882744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9307.751293] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9318.787861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9318.826066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.299346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9320.344698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9320.420725] Lustre: lustre-MDT0000: new disk, initializing [ 9320.594427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9320.623535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9324.986968] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9336.969808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.155391] Lustre: 252814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9337.242203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9337.249394] Lustre: Skipped 1 previous similar message [ 9337.329279] Lustre: lustre-MDT0001: new disk, initializing [ 9337.416471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9337.442078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9337.458820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9342.784038] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9348.407304] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9357.571609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.040067] Lustre: lustre-OST0000: new disk, initializing [ 9358.044975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9358.051354] Lustre: 254447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9358.223772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.787529] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9367.194508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9367.212665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9367.364117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9379.032729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.372840] Lustre: lustre-OST0001: new disk, initializing [ 9379.387904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9379.405556] Lustre: 255318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.712276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9387.624754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9387.636197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9387.733325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9388.813146] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9401.481292] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9410.966769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9428.969631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.980525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.984392] Lustre: Skipped 1 previous similar message [ 9428.999448] Lustre: Skipped 3 previous similar messages [ 9433.480457] Lustre: server umount lustre-MDT0000 complete [ 9434.081556] LustreError: 253834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.114615] LustreError: 253834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9437.725776] LustreError: 252807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756873 with bad export cookie 5090613002957126816 [ 9437.735779] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9437.742905] LustreError: 252807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9438.145560] Lustre: server umount lustre-MDT0001 complete [ 9452.881397] Lustre: server umount lustre-OST0000 complete [ 9455.392301] Lustre: 250320:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756875/real 1786756875] req@ffff9bfb4b758000 x1873550184106112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786756891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9455.438298] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.454557] Lustre: Skipped 2 previous similar messages [ 9457.063193] Lustre: server umount lustre-OST0001 complete [ 9472.267356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9485.513340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9497.065828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9507.410447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9532.448327] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9543.883850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.925594] Lustre: lustre-MDT0000: reset Object Index mappings [ 9544.462198] LustreError: 260330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.517708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.800927] LustreError: 260331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9549.964062] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9554.915359] LustreError: 260330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.623365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.641736] Lustre: lustre-MDT0001: reset Object Index mappings [ 9558.998181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9564.028095] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9567.788843] Lustre: 261473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9575.037582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9575.378807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.112906] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9583.606162] LustreError: 261828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.760812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9591.207371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9596.952656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9598.548677] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9606.517174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9610.874538] Lustre: 263347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9617.377446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.384433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.396478] Lustre: Skipped 1 previous similar message [ 9618.195200] Lustre: server umount lustre-MDT0000 complete [ 9622.497753] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757058 with bad export cookie 5090613002957133606 [ 9622.499936] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9622.515600] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9622.559760] LustreError: 263113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.568913] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.588926] LustreError: 263113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9622.596184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9622.625592] Lustre: Skipped 3 previous similar messages [ 9622.664041] Lustre: Skipped 4 previous similar messages [ 9623.176861] Lustre: server umount lustre-MDT0001 complete [ 9628.805050] Lustre: server umount lustre-OST0000 complete [ 9633.014837] Lustre: server umount lustre-OST0001 complete [ 9654.419518] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:24:48 (1786757088) [ 9672.882625] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9688.429853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9731.502247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.957359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9732.013905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9732.244708] Lustre: lustre-MDT0000: new disk, initializing [ 9732.407896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9732.411870] Lustre: Skipped 1 previous similar message [ 9732.468431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9739.891253] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9751.087759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.249184] Lustre: 267617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9751.299799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9751.303906] Lustre: Skipped 1 previous similar message [ 9751.378105] Lustre: lustre-MDT0001: new disk, initializing [ 9751.526749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9751.548018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9756.374035] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9767.185601] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9774.557335] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9783.456637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.779961] Lustre: lustre-OST0000: new disk, initializing [ 9783.784378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9783.793457] Lustre: 269590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.983236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9784.992688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9785.103394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9790.297975] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9799.672453] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9804.286325] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9804.581667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9810.011929] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9810.288868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.034461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.047986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.061640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.070459] Lustre: Skipped 1 previous similar message [ 9818.482338] Lustre: server umount lustre-OST0000 complete [ 9823.203135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9823.210770] LustreError: Skipped 1 previous similar message [ 9823.227082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9823.237153] Lustre: Skipped 1 previous similar message [ 9826.275341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.292217] Lustre: Skipped 1 previous similar message [ 9829.101301] Lustre: server umount lustre-MDT0000 complete [ 9831.407294] LustreError: 267631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.436395] LustreError: 267631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9833.470209] LustreError: 267610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757269 with bad export cookie 5090613002957135699 [ 9833.478193] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9833.481931] LustreError: 267610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9833.900952] Lustre: server umount lustre-MDT0001 complete [ 9856.480582] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9869.460590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.111398] LustreError: 272525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9870.174988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.181175] Lustre: Skipped 2 previous similar messages [ 9874.644262] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9883.435898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.722742] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9893.949941] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9903.296201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9910.567654] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9920.084451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.326940] Lustre: lustre-OST0001: new disk, initializing [ 9920.345404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9920.362697] Lustre: 274837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9922.417300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9922.443169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9922.705974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9927.653843] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9938.397896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9942.380261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9947.730261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9947.865389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9955.304112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9955.317477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.351993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9955.365599] Lustre: Skipped 2 previous similar messages [ 9960.456189] Lustre: server umount lustre-MDT0000 complete [ 9963.511809] LustreError: 273454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.556960] LustreError: 273454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9965.595926] LustreError: 273661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757401 with bad export cookie 5090613002957136581 [ 9965.614865] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9965.618098] LustreError: 273661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9966.229582] Lustre: server umount lustre-MDT0001 complete [ 9979.243355] Lustre: server umount lustre-OST0000 complete [ 9994.223172] Lustre: server umount lustre-OST0001 complete [10013.691414] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10029.264185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.917604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.920701] Lustre: Skipped 3 previous similar messages [10036.940845] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10051.083988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.796540] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10059.460653] Lustre: 279037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10067.317562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10074.240490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10075.912684] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10086.258928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.627461] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10094.006819] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10102.578224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10107.143851] Lustre: 280915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10112.048600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10112.059459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10121.603135] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:32:36 (1786757556) [10137.570841] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.573655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10137.577091] Lustre: Skipped 6 previous similar messages [10137.604814] Lustre: Skipped 7 previous similar messages [10151.391254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10151.764036] Lustre: server umount lustre-MDT0000 complete [10152.933519] LustreError: 279120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.945483] LustreError: 279120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10155.269042] LustreError: 278639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757591 with bad export cookie 5090613002957143371 [10155.276247] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10155.281228] LustreError: 278639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10161.743400] Lustre: server umount lustre-MDT0001 complete [10172.433307] Lustre: server umount lustre-OST0000 complete [10176.058983] Lustre: server umount lustre-OST0001 complete [10184.450645] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [10196.780506] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10249.526169] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10264.107647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.425932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.454644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10264.552373] Lustre: lustre-MDT0000: new disk, initializing [10264.824496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10271.646322] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10285.459833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.709964] Lustre: 285159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10286.122863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10286.131976] Lustre: Skipped 4 previous similar messages [10286.221763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10291.871552] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10297.175908] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10305.151863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.579574] Lustre: 286794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10307.230700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10314.391066] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10325.826573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10325.982898] Lustre: 287666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10327.932481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10327.947565] Lustre: Skipped 1 previous similar message [10328.083596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10331.876721] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10343.257961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10368.007842] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:36:42 (1786757802) [10372.656836] Lustre: server umount lustre-MDT0000 complete [10373.089684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10373.114809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.147320] Lustre: Skipped 2 previous similar messages [10377.015617] LustreError: 286804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757812 with bad export cookie 5090613002957150798 [10377.017889] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10377.042407] LustreError: 286804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10377.588549] Lustre: server umount lustre-MDT0001 complete [10391.325857] Lustre: server umount lustre-OST0000 complete [10405.620218] Lustre: server umount lustre-OST0001 complete [10415.347852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.853467] LustreError: 289913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.874485] LustreError: 289913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10419.916173] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10428.240508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.920931] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10440.930213] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10447.344165] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10454.884771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.547425] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10472.906953] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10496.187136] loop1: detected capacity change from 0 to 204800000 [10496.579564] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10496.597164] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10496.624962] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10502.287672] loop1: detected capacity change from 0 to 204800000 [10509.061642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.228716] LustreError: 289903:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30089: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10509.265201] LustreError: 289903:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30089 [10509.286175] LustreError: 289903:0:(mgs_handler.c:517:mgs_target_reg()) fs30089-MDT0001: Error checking target index: rc = -13 [10509.301287] LustreError: fs30089-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10509.308351] LustreError: 293633:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10509.316443] LustreError: 293633:0:(tgt_mount.c:2129:server_put_super()) no obd fs30089-MDT0001 [10509.327192] LustreError: 293633:0:(tgt_mount.c:116:server_deregister_mount()) fs30089-MDT0001 not registered [10509.380147] Lustre: server umount fs30089-MDT0001 complete [10509.382250] LustreError: 293633:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10520.033680] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.221600] Lustre: Setting parameter fs30089-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30089-MDT0001 [10520.227899] Lustre: Skipped 1 previous similar message [10520.266845] Lustre: srv-fs30089-MDT0001: No data found on store. Initialize space. [10520.274871] Lustre: Skipped 4 previous similar messages [10520.384359] Lustre: fs30089-MDT0001: new disk, initializing [10520.387980] Lustre: Skipped 3 previous similar messages [10520.542254] Lustre: srv-fs30089-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10520.557464] Lustre: 294101:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30089-MDT0001-mdtlov: can't create new llog: rc = -115 [10520.581382] LustreError: 294101:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30089-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10526.697873] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10534.889883] Lustre: server umount fs30089-MDT0001 complete [10540.527151] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10549.167467] loop1: detected capacity change from 0 to 204800000 [10555.890658] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10556.400657] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10556.791474] Lustre: 295669:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30089-MDT0001-mdtlov: can't create new llog: rc = -115 [10556.803037] LustreError: 295669:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30089-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10561.258362] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10565.999777] Lustre: server umount fs30089-MDT0001 complete [10575.773848] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [10575.779943] Lustre: Skipped 12 previous similar messages [10576.356512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.368455] Lustre: Skipped 5 previous similar messages [10581.670072] Lustre: server umount lustre-MDT0000 complete [10590.084520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.273777] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10594.737807] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10596.359811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10602.735109] loop1: detected capacity change from 0 to 204800000 [10608.181859] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.543320] Lustre: 298114:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30089-MDT0001-mdtlov: can't create new llog: rc = -115 [10608.547634] LustreError: 298114:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30089-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10613.536597] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10618.579448] Lustre: server umount fs30089-MDT0001 complete [10641.889199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10641.899564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10641.906054] Lustre: Skipped 7 previous similar messages [10647.818561] Lustre: server umount lustre-OST0000 complete [10661.053492] Lustre: server umount lustre-MDT0000 complete [10666.318144] LustreError: 290660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758102 with bad export cookie 5090613002957153479 [10666.322246] LustreError: 290660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10666.325147] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10666.779127] Lustre: server umount lustre-MDT0001 complete [10681.903885] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [10686.269816] Key type lgssc unregistered [10686.716940] LNet: 300462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10686.725464] LNetError: 300462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10686.764955] LNet: Removed LNI 192.168.201.110@tcp [10688.001240] Key type .llcrypt unregistered [10688.009127] Key type ._llcrypt unregistered [10716.488288] Key type ._llcrypt registered [10716.498810] Key type .llcrypt registered [10716.716200] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10718.076852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10718.423820] alg: No test for adler32 (adler32-zlib) [10719.699128] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [10719.940799] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [10721.615177] Key type lgssc registered [10722.885860] Lustre: Echo OBD driver; http://www.lustre.org/ [10730.651645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10730.679775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10732.276100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10737.045840] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10742.755652] LustreError: 301737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.768259] LustreError: 301737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10746.022443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.652594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10750.707958] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10758.228309] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10764.226534] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10770.697593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.368863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10772.404816] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.514364] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.548520] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10778.308763] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10782.712295] LustreError: 303876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.733462] LustreError: 303876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10786.330815] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10787.818702] LustreError: 303876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.839025] LustreError: 303876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10798.067426] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.080329] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10807.342191] loop1: detected capacity change from 0 to 204800000 [10807.641401] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10807.656618] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10807.691587] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10813.139859] loop1: detected capacity change from 0 to 204800000 [10814.837804] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.862421] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10818.988177] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.137301] Lustre: fs31157-OST0000: new disk, initializing [10819.140459] Lustre: srv-fs31157-OST0000: No data found on store. Initialize space. [10819.144951] Lustre: 305474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10819.207768] Lustre: fs31157-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10825.877439] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10834.158659] Lustre: server umount fs31157-OST0000 complete [10846.192437] loop1: detected capacity change from 0 to 204800000 [10849.258230] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.293052] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10850.910255] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.121260] Lustre: fs31157-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.474822] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10859.503815] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10863.275629] loop2: detected capacity change from 0 to 204800000 [10863.549031] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10863.567777] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10863.590246] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.289198] loop2: detected capacity change from 0 to 204800000 [10874.701817] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.817851] LustreError: fs31157-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10874.823574] LustreError: fs31157-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10874.834382] LustreError: 308512:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10874.841730] LustreError: 308512:0:(tgt_mount.c:2129:server_put_super()) no obd fs31157-OST0001 [10874.846582] LustreError: 308512:0:(tgt_mount.c:116:server_deregister_mount()) fs31157-OST0001 not registered [10874.894900] Lustre: server umount fs31157-OST0001 complete [10874.900575] LustreError: 308512:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10886.371992] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10886.471467] Lustre: fs31157-OST0001: new disk, initializing [10886.473897] Lustre: srv-fs31157-OST0001: No data found on store. Initialize space. [10886.478139] Lustre: 309064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10886.544357] Lustre: fs31157-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10892.530689] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10900.680943] Lustre: server umount fs31157-OST0000 complete [10911.542757] Lustre: server umount fs31157-OST0001 complete [10915.817023] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.841973] LustreError: 303571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10923.849426] loop1: detected capacity change from 0 to 204800000 [10929.567374] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10929.828983] Lustre: fs31157-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10936.429181] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10943.658531] loop2: detected capacity change from 0 to 204800000 [10950.068177] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10957.105330] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10967.151114] Lustre: server umount fs31157-OST0000 complete [10978.344503] Lustre: server umount fs31157-OST0001 complete [10997.771331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10997.785338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10997.814206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10997.818334] LustreError: Skipped 2 previous similar messages [10997.838026] Lustre: Skipped 1 previous similar message [10999.455362] Lustre: server umount lustre-OST0000 complete [11007.460577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11007.476581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11007.499229] Lustre: Skipped 1 previous similar message [11007.510447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.013787] Lustre: server umount lustre-MDT0000 complete [11013.786978] LustreError: 301718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758449 with bad export cookie 14752993025984531660 [11013.801110] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11013.812818] LustreError: 301718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11014.262517] Lustre: server umount lustre-MDT0001 complete [11024.223932] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11026.725402] Key type lgssc unregistered [11027.027750] LNet: 314989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11027.035882] LNetError: 314989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11027.052393] LNet: Removed LNI 192.168.201.110@tcp [11027.914162] Key type .llcrypt unregistered [11027.918962] Key type ._llcrypt unregistered [11056.667894] Key type ._llcrypt registered [11056.669479] Key type .llcrypt registered [11056.780411] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11070.773156] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11072.215160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11072.253591] alg: No test for adler32 (adler32-zlib) [11073.260993] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11073.425935] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11075.087554] Key type lgssc registered [11076.068893] Lustre: Echo OBD driver; http://www.lustre.org/ [11107.356265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11107.379825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11108.558747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11108.580602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11108.646985] Lustre: lustre-MDT0000: new disk, initializing [11108.709771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11108.721249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11112.832901] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11123.768705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.899523] Lustre: 318850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11123.962911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11123.966933] Lustre: Skipped 1 previous similar message [11124.029981] Lustre: lustre-MDT0001: new disk, initializing [11124.105212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11124.128895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11124.144641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11128.432602] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11136.895806] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11141.377859] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11147.477533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.706148] Lustre: lustre-OST0000: new disk, initializing [11147.720705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11147.738541] Lustre: 320817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11147.818093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11153.687245] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11154.511573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11154.518481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11154.667282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11161.309215] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11162.905270] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841 [11163.667401] Lustre: Unmounted lustre-client [11173.345139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.371085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.386307] Lustre: Skipped 1 previous similar message [11177.636204] Lustre: server umount lustre-OST0000 complete [11180.512608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11180.517185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11180.528320] Lustre: Skipped 1 previous similar message [11180.536721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11183.584669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11183.585863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.531529] Lustre: server umount lustre-MDT0000 complete [11190.518187] LustreError: 318841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758626 with bad export cookie 4525002601005272822 [11190.539860] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11190.921111] Lustre: server umount lustre-MDT0001 complete [11200.993645] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11203.520943] Key type lgssc unregistered [11216.802205] LNet: 1 local NIs in recovery (showing 1): 192.168.201.110@tcp [11217.123175] LNet: 322978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11217.137017] LNetError: 322978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11293.998120] LNet: Removed LNI 192.168.201.110@tcp [11294.993472] Key type .llcrypt unregistered [11294.997126] Key type ._llcrypt unregistered [11313.963235] Key type ._llcrypt registered [11313.965234] Key type .llcrypt registered [11314.052275] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11327.798363] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11328.836533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11328.879124] alg: No test for adler32 (adler32-zlib) [11329.951099] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11330.140724] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11331.783936] Key type lgssc registered [11332.891654] Lustre: Echo OBD driver; http://www.lustre.org/ [11362.628944] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:53:17 (1786758797) [11368.758093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11368.806224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.003777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11370.108113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11370.215723] Lustre: lustre-MDT0000: new disk, initializing [11370.304768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11370.320673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11375.382533] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11386.323269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.456966] Lustre: 326194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11386.529663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11386.536678] Lustre: Skipped 1 previous similar message [11386.615706] Lustre: lustre-MDT0001: new disk, initializing [11386.687105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11386.709369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11386.713768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11390.279113] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11399.405913] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11405.308352] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11412.604187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11412.830819] Lustre: lustre-OST0000: new disk, initializing [11412.837148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11412.844236] Lustre: 328163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11412.954243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11419.348587] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11419.693758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11419.708890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11419.846714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11428.587713] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11445.730924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11445.733821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11445.752709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11445.777247] Lustre: Skipped 1 previous similar message [11447.076635] Lustre: server umount lustre-OST0000 complete [11450.849195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.862410] Lustre: Skipped 1 previous similar message [11450.872205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11455.968685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11455.981614] Lustre: Skipped 2 previous similar messages [11457.253583] Lustre: server umount lustre-MDT0000 complete [11460.526791] LustreError: 326184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758896 with bad export cookie 6606076108432438537 [11460.544206] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11460.889126] Lustre: server umount lustre-MDT0001 complete [11475.036547] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11478.032723] Key type lgssc unregistered [11478.356712] LNet: 330072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11478.366869] LNetError: 330072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11479.398077] LNet: Removed LNI 192.168.201.110@tcp [11480.465401] Key type .llcrypt unregistered [11480.467169] Key type ._llcrypt unregistered [11502.788748] Key type ._llcrypt registered [11502.791121] Key type .llcrypt registered [11502.874294] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11503.921373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11503.991351] alg: No test for adler32 (adler32-zlib) [11505.059788] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11505.314975] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11506.967204] Key type lgssc registered [11507.945153] Lustre: Echo OBD driver; http://www.lustre.org/ [11515.095776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11515.115481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.645647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11520.709570] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11526.119493] LustreError: 331349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.151764] LustreError: 331349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11528.528953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.847404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11532.083830] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11538.512313] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11543.815604] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11548.898822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.165674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11553.835958] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11559.811523] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11573.445855] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11574.797545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11579.874025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11579.888954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.913750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11581.838573] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.10@tcp (stopping) [11581.926524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.937169] Lustre: Skipped 1 previous similar message [11584.597889] Lustre: server umount lustre-MDT0000 complete [11586.935289] LustreError: 331344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.519308] LustreError: 331329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759023 with bad export cookie 1107587181052646291 [11587.532427] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11587.536043] LustreError: 331329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11592.071143] LustreError: 331344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.098129] LustreError: 331344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11592.108036] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [11592.118133] Lustre: Skipped 3 previous similar messages [11592.161893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11593.919994] Lustre: server umount lustre-MDT0001 complete [11597.812784] Lustre: server umount lustre-OST0000 complete [11606.646825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.914601] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11615.073676] Lustre: 335526:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.110@tcp [11617.052560] Lustre: server umount MGS complete [11625.027039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.342538] LustreError: 336098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.360792] LustreError: 336098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11625.408888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11629.670331] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11630.563141] LustreError: 336099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.029601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.385276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11644.186570] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11652.820625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11653.178066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11659.022463] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11664.841556] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11665.909075] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11671.070930] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11676.057786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11681.264461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11681.287273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.322468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.334351] Lustre: Skipped 1 previous similar message [11683.152146] Lustre: server umount lustre-OST0000 complete [11690.467185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11690.474182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.494958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.893963] Lustre: server umount lustre-MDT0000 complete [11696.114139] LustreError: 336099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.146835] LustreError: 336099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11698.464178] LustreError: 336078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759134 with bad export cookie 1107587181052647131 [11698.470358] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11698.478424] LustreError: 336078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11709.106165] Lustre: server umount lustre-MDT0001 complete [11722.741679] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11726.361983] Key type lgssc unregistered [11726.686675] LNet: 339539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11726.691994] LNetError: 339539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11726.708435] LNet: Removed LNI 192.168.201.110@tcp [11727.524776] Key type .llcrypt unregistered [11727.527390] Key type ._llcrypt unregistered [11748.829310] Key type ._llcrypt registered [11748.831886] Key type .llcrypt registered [11748.901289] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11766.946280] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11768.043766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11768.250539] alg: No test for adler32 (adler32-zlib) [11769.334313] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11769.604747] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11771.263639] Key type lgssc registered [11772.643599] Lustre: Echo OBD driver; http://www.lustre.org/ [11807.100488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11807.138137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.372281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11808.400754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11808.481220] Lustre: lustre-MDT0000: new disk, initializing [11808.539945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.563592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11813.250198] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11825.431313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11825.693440] Lustre: 342701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11825.906916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11825.933261] Lustre: Skipped 1 previous similar message [11826.042290] Lustre: lustre-MDT0001: new disk, initializing [11826.143029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.205135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11826.218738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11830.982976] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11840.467672] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11846.226964] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11854.065831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11854.394718] Lustre: lustre-OST0000: new disk, initializing [11854.398512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11854.406651] Lustre: 344686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11854.493438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.433628] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11864.145986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11864.165785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11864.314579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11872.218431] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11877.792287] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11878.033082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11882.788911] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11882.968653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11885.024196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11885.032656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.036167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11885.051910] Lustre: Skipped 1 previous similar message [11890.156579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11890.171823] Lustre: Skipped 2 previous similar messages [11895.269368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11895.273922] Lustre: Skipped 1 previous similar message [11899.359233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11899.645863] Lustre: server umount lustre-OST0000 complete [11905.505284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11905.527612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11905.538038] Lustre: Skipped 1 previous similar message [11910.629729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11910.641276] Lustre: Skipped 1 previous similar message [11910.871507] Lustre: server umount lustre-MDT0000 complete [11916.331609] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759352 with bad export cookie 10106938258712838268 [11916.346509] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11916.354449] LustreError: 342695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11916.762211] Lustre: server umount lustre-MDT0001 complete [11925.776956] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:02:40 (1786759360) [11943.351245] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11955.657932] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11995.765077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11995.963803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11996.000298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11996.105362] Lustre: lustre-MDT0000: new disk, initializing [11996.195537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11996.217272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12001.379401] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12012.625231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.788336] Lustre: 349448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12012.849223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12012.854826] Lustre: Skipped 1 previous similar message [12012.938497] Lustre: lustre-MDT0001: new disk, initializing [12013.038885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12013.103087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12013.119559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12017.462806] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12027.835634] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12033.347364] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12039.574956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12039.900858] Lustre: lustre-OST0000: new disk, initializing [12039.905363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12039.911750] Lustre: 351414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12040.013228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12045.913679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12045.925644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12046.183400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12047.453518] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12057.160594] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12087.269342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.276393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.278482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12087.281564] Lustre: Skipped 2 previous similar messages [12087.308536] Lustre: Skipped 1 previous similar message [12089.857530] Lustre: server umount lustre-OST0000 complete [12094.944507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12094.952827] LustreError: Skipped 1 previous similar message [12094.958109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.314481] Lustre: server umount lustre-MDT0000 complete [12102.627042] LustreError: 349457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.640398] LustreError: 349457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12103.293490] LustreError: 350242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759539 with bad export cookie 10106938258712839150 [12103.300888] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12103.320266] LustreError: 350242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12103.754782] Lustre: server umount lustre-MDT0001 complete [12115.025826] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12117.841412] Key type lgssc unregistered [12118.098475] LNet: 353567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12118.109441] LNetError: 353567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12118.122949] LNet: Removed LNI 192.168.201.110@tcp [12118.963349] Key type .llcrypt unregistered [12118.965221] Key type ._llcrypt unregistered [12215.238613] Key type ._llcrypt registered [12215.243381] Key type .llcrypt registered [12215.393574] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12215.690874] Key type .llcrypt unregistered [12215.692058] Key type ._llcrypt unregistered [12297.538328] Key type ._llcrypt registered [12297.542691] Key type .llcrypt registered [12297.658256] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12297.952737] Key type .llcrypt unregistered [12297.956480] Key type ._llcrypt unregistered [12330.919750] Key type ._llcrypt registered [12330.926480] Key type .llcrypt registered [12331.032722] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12331.352679] Key type .llcrypt unregistered [12331.354934] Key type ._llcrypt unregistered [12368.931395] Key type ._llcrypt registered [12368.938451] Key type .llcrypt registered [12369.088770] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12369.531457] Key type .llcrypt unregistered [12369.537642] Key type ._llcrypt unregistered [12449.152092] Key type ._llcrypt registered [12449.159446] Key type .llcrypt registered [12449.300780] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12449.672290] Key type .llcrypt unregistered [12449.676681] Key type ._llcrypt unregistered [12479.066598] Key type ._llcrypt registered [12479.071936] Key type .llcrypt registered [12479.208815] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12479.496340] Key type .llcrypt unregistered [12479.498623] Key type ._llcrypt unregistered [12507.546962] Key type ._llcrypt registered [12507.548945] Key type .llcrypt registered [12507.703046] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12508.199516] Key type .llcrypt unregistered [12508.201631] Key type ._llcrypt unregistered [12541.740411] Key type ._llcrypt registered [12541.742098] Key type .llcrypt registered [12541.938934] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12542.453527] Key type .llcrypt unregistered [12542.463807] Key type ._llcrypt unregistered [12581.903928] Key type ._llcrypt registered [12581.912891] Key type .llcrypt registered [12582.033587] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12582.364431] Key type .llcrypt unregistered [12582.366775] Key type ._llcrypt unregistered [12608.325730] Key type ._llcrypt registered [12608.328437] Key type .llcrypt registered [12608.437127] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12608.677320] Key type .llcrypt unregistered [12608.679108] Key type ._llcrypt unregistered [12663.877932] Key type ._llcrypt registered [12663.880917] Key type .llcrypt registered [12664.010745] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12664.306255] Key type .llcrypt unregistered [12664.311253] Key type ._llcrypt unregistered [12695.779160] Key type ._llcrypt registered [12695.783560] Key type .llcrypt registered [12695.890891] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12696.165398] Key type .llcrypt unregistered [12696.166888] Key type ._llcrypt unregistered [12726.220606] Key type ._llcrypt registered [12726.224211] Key type .llcrypt registered [12726.382549] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12726.743285] Key type .llcrypt unregistered [12726.745057] Key type ._llcrypt unregistered [12785.524708] Key type ._llcrypt registered [12785.527070] Key type .llcrypt registered [12785.637353] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12799.919260] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12800.957347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12801.069580] alg: No test for adler32 (adler32-zlib) [12802.157866] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [12802.329534] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [12803.959164] Key type lgssc registered [12805.174180] Lustre: Echo OBD driver; http://www.lustre.org/ [12837.506101] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12837.550472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12838.753406] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12838.795191] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12838.919347] Lustre: tst250fs-MDT0000: new disk, initializing [12838.977526] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12839.001223] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12843.809423] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12853.963650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.080648] Lustre: 373906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12854.129095] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12854.137278] Lustre: Skipped 1 previous similar message [12854.228594] Lustre: tst250fs-MDT0001: new disk, initializing [12854.393624] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.428956] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12854.435387] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12858.594875] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12863.895894] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12870.754958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12871.101375] Lustre: tst250fs-OST0000: new disk, initializing [12871.106655] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12871.113885] Lustre: 375540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12871.241129] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12871.591939] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12871.605930] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12871.734877] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12876.743363] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12889.540741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.829137] Lustre: tst250fs-OST0001: new disk, initializing [12889.838737] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12889.858364] Lustre: 376413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12890.055395] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12895.475567] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12896.351983] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12896.364306] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12896.461919] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12899.899641] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12902.084474] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12902.087199] Lustre: Skipped 2 previous similar messages [12920.804177] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12920.813612] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12920.833491] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12922.343883] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.346899] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12922.377081] Lustre: Skipped 2 previous similar messages [12922.387031] Lustre: Skipped 3 previous similar messages [12923.907663] Lustre: server umount tst250fs-MDT0000 complete [12927.806242] LustreError: 373898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760363 with bad export cookie 16989769922945283718 [12927.819314] LustreError: 373898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12927.824815] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12928.071531] Lustre: server umount tst250fs-MDT0001 complete [12942.021650] Lustre: server umount tst250fs-OST0000 complete [12955.574751] Lustre: server umount tst250fs-OST0001 complete [12972.461447] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12980.892702] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [13009.927710] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:20:44 (1786760444) [13011.658229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13026.688385] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [13035.298507] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [13061.136774] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12750 sec ========= 22:21:35 (1786760495) [13063.035343] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:21:37 (1786760497) === [13067.459921] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:21:41 (1786760501) === [13107.919887] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13111.490843] Key type lgssc unregistered [13111.870686] LNet: 385555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13111.878370] LNetError: 385555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13111.902731] LNet: Removed LNI 192.168.201.110@tcp [13113.053142] Key type .llcrypt unregistered [13113.057372] Key type ._llcrypt unregistered