[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 960610093 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009143] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011064] *** VALIDATE selinux *** [ 0.019630] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023761] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026121] *** VALIDATE tmpfs *** [ 0.027429] *** VALIDATE proc *** [ 0.028266] *** VALIDATE cgroup *** [ 0.029016] *** VALIDATE cgroup2 *** [ 0.030291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033029] Spectre V2 : User space: Vulnerable [ 0.034009] Speculative Store Bypass: Vulnerable [ 0.037240] debug: unmapping init [mem 0xffffffff9cc59000-0xffffffff9cc60fff] [ 0.039253] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042010] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044012] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046017] ... fixed-purpose events: 3 [ 0.047017] ... event mask: 000000070000000f [ 0.048317] rcu: Hierarchical SRCU implementation. [ 0.050690] smp: Bringing up secondary CPUs ... [ 0.051670] x86: Booting SMP configuration: [ 0.052026] .... node #0, CPUs: #1 #2 #3 [ 0.062147] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.113472] node 0 deferred pages initialised in 46ms [ 0.118008] devtmpfs: initialized [ 0.119328] x86/mm: Memory block size: 128MB [ 0.121968] gcov: version magic: 0x41383552 [ 0.124107] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.125083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.126630] pinctrl core: initialized pinctrl subsystem [ 0.128137] [ 0.129012] ************************************************************* [ 0.130015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131016] ** ** [ 0.132017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133015] ** ** [ 0.134030] ** This means that this kernel is built to expose internal ** [ 0.135015] ** IOMMU data structures, which may compromise security on ** [ 0.136012] ** your system. ** [ 0.137069] ** ** [ 0.138013] ** If you see this message and you are not debugging the ** [ 0.139014] ** kernel, report this immediately to your vendor! ** [ 0.140012] ** ** [ 0.141013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142012] ************************************************************* [ 0.143526] NET: Registered protocol family 16 [ 0.144389] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.147531] cpuidle: using governor menu [ 0.148825] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.150484] PCI: Using configuration type 1 for base access [ 0.153132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.164237] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.166046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.170062] cryptd: max_cpu_qlen set to 1000 [ 0.171897] ACPI: Added _OSI(Module Device) [ 0.174012] ACPI: Added _OSI(Processor Device) [ 0.175008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.177009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.181716] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188251] ACPI: Interpreter enabled [ 0.189098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190009] ACPI: Using IOAPIC for interrupt routing [ 0.191077] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.192401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.202786] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.203032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.204015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.205074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.207417] acpiphp: Slot [2] registered [ 0.208146] acpiphp: Slot [5] registered [ 0.209119] acpiphp: Slot [6] registered [ 0.210115] acpiphp: Slot [7] registered [ 0.211085] acpiphp: Slot [8] registered [ 0.212082] acpiphp: Slot [9] registered [ 0.213102] acpiphp: Slot [10] registered [ 0.214093] acpiphp: Slot [3] registered [ 0.215097] acpiphp: Slot [4] registered [ 0.216095] acpiphp: Slot [11] registered [ 0.217096] acpiphp: Slot [12] registered [ 0.218077] acpiphp: Slot [13] registered [ 0.219076] acpiphp: Slot [14] registered [ 0.220086] acpiphp: Slot [15] registered [ 0.221073] acpiphp: Slot [16] registered [ 0.222066] acpiphp: Slot [17] registered [ 0.223075] acpiphp: Slot [18] registered [ 0.224054] acpiphp: Slot [19] registered [ 0.225065] acpiphp: Slot [20] registered [ 0.226068] acpiphp: Slot [21] registered [ 0.227062] acpiphp: Slot [22] registered [ 0.228056] acpiphp: Slot [23] registered [ 0.229070] acpiphp: Slot [24] registered [ 0.230055] acpiphp: Slot [25] registered [ 0.231060] acpiphp: Slot [26] registered [ 0.232073] acpiphp: Slot [27] registered [ 0.233070] acpiphp: Slot [28] registered [ 0.234078] acpiphp: Slot [29] registered [ 0.235106] acpiphp: Slot [30] registered [ 0.236126] acpiphp: Slot [31] registered [ 0.237048] PCI host bridge to bus 0000:00 [ 0.238014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.241014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.242016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.243017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.244141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.245907] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.247281] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.253012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.257049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.258013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.259012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.260010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.261549] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.262799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.263040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.264838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.267015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.276014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.279011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.282392] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.291015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.301022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.324014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.343108] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.354014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.362014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.384017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439170] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.448015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.454016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.480422] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.487018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.492016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.507015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.517706] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.528016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.538014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.560017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.568626] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.575013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.579015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.592015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.599535] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.600236] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.602225] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.604279] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.605110] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.609125] iommu: Default domain type: Passthrough [ 0.611597] SCSI subsystem initialized [ 0.613145] ACPI: bus type USB registered [ 0.614112] usbcore: registered new interface driver usbfs [ 0.617085] usbcore: registered new interface driver hub [ 0.619073] usbcore: registered new device driver usb [ 0.621258] pps_core: LinuxPPS API ver. 1 registered [ 0.622007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.625044] PTP clock support registered [ 0.627142] EDAC MC: Ver: 3.0.0 [ 0.628662] PCI: Using ACPI for IRQ routing [ 0.632059] NetLabel: Initializing [ 0.633006] NetLabel: domain hash size = 128 [ 0.634008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.635000] NetLabel: unlabeled traffic allowed by default [ 0.636238] vgaarb: loaded [ 0.638282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.641014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.647130] clocksource: Switched to clocksource kvm-clock [ 0.751996] VFS: Disk quotas dquot_6.6.0 [ 0.753615] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.756337] *** VALIDATE ramfs *** [ 0.757695] *** VALIDATE hugetlbfs *** [ 0.759572] pnp: PnP ACPI init [ 0.762451] pnp: PnP ACPI: found 6 devices [ 0.778017] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.781208] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.782534] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.783943] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.786271] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.788326] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.790264] NET: Registered protocol family 2 [ 0.792080] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.795601] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.798070] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802310] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.805559] TCP: Hash tables configured (established 65536 bind 65536) [ 0.808650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811089] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814140] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816686] NET: Registered protocol family 1 [ 0.819991] RPC: Registered named UNIX socket transport module. [ 0.822497] RPC: Registered udp transport module. [ 0.824353] RPC: Registered tcp transport module. [ 0.826734] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.828740] NET: Registered protocol family 44 [ 0.829773] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.831688] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.833073] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.835246] PCI: CLS 0 bytes, default 64 [ 0.836864] Unpacking initramfs... [ 2.580358] debug: unmapping init [mem 0xffff96c9bcc54000-0xffff96c9bffbffff] [ 2.583903] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.585317] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.589314] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.115108] Initialise system trusted keyrings [ 3.116452] Key type blacklist registered [ 3.117724] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.126235] zbud: loaded [ 3.128732] *** VALIDATE nfs *** [ 3.129439] *** VALIDATE nfs4 *** [ 3.130493] pstore: using deflate compression [ 3.133607] Platform Keyring initialized [ 3.235350] NET: Registered protocol family 38 [ 3.236442] Key type asymmetric registered [ 3.237286] Asymmetric key parser 'x509' registered [ 3.238367] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.240431] io scheduler mq-deadline registered [ 3.241702] io scheduler kyber registered [ 3.243469] io scheduler bfq registered [ 3.245182] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.248955] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.251727] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.254495] ACPI: Power Button [PWRF] [ 3.264417] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.271700] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.285582] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.292561] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.314458] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.342065] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.370871] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.375889] Non-volatile memory driver v1.3 [ 3.377560] Linux agpgart interface v0.103 [ 3.402887] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.405620] vda: detected capacity change from 0 to 74698752 [ 3.418221] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.420602] vdb: detected capacity change from 0 to 1073741824 [ 3.432667] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.435643] vdc: detected capacity change from 0 to 2621440000 [ 3.446996] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.449029] vdd: detected capacity change from 0 to 2621440000 [ 3.461053] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.463319] vde: detected capacity change from 0 to 4294967296 [ 3.474970] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.477541] vdf: detected capacity change from 0 to 4294967296 [ 3.483461] libphy: Fixed MDIO Bus: probed [ 3.487991] usbcore: registered new interface driver usbserial_generic [ 3.490216] usbserial: USB Serial support registered for generic [ 3.491991] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.495437] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.496958] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.500167] mousedev: PS/2 mouse device common for all mice [ 3.503847] rtc_cmos 00:05: RTC can wake from S4 [ 3.505779] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.507283] rtc_cmos 00:05: registered as rtc0 [ 3.512777] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.513369] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.515116] intel_pstate: CPU model not supported [ 3.520550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.523073] hid: raw HID events driver (C) Jiri Kosina [ 3.525589] usbcore: registered new interface driver usbhid [ 3.527624] usbhid: USB HID core driver [ 3.529244] drop_monitor: Initializing network drop monitor service [ 3.531446] Initializing XFRM netlink socket [ 3.533370] NET: Registered protocol family 10 [ 3.535865] Segment Routing with IPv6 [ 3.537583] NET: Registered protocol family 17 [ 3.538857] mpls_gso: MPLS GSO support [ 3.543601] RAS: Correctable Errors collector initialized. [ 3.545237] AVX version of gcm_enc/dec engaged. [ 3.546390] AES CTR mode by8 optimization enabled [ 3.602767] sched_clock: Marking stable (3602738326, 0)->(4641339459, -1038601133) [ 3.604852] registered taskstats version 1 [ 3.606265] Loading compiled-in X.509 certificates [ 3.607841] zswap: loaded using pool lzo/zbud [ 3.624817] Key type big_key registered [ 3.633454] Key type encrypted registered [ 3.634811] ima: No TPM chip found, activating TPM-bypass! [ 3.636055] ima: Allocated hash algorithm: sha1 [ 3.637097] ima: No architecture policies found [ 3.638306] evm: Initialising EVM extended attributes: [ 3.639520] evm: security.selinux [ 3.640362] evm: security.ima [ 3.641200] evm: security.capability [ 3.642076] evm: HMAC attrs: 0x1 [ 3.644213] rtc_cmos 00:05: setting system clock to 2026-08-08 08:23:13 UTC (1786177393) [ 3.649429] debug: unmapping init [mem 0xffffffff9dc03000-0xffffffff9ddfffff] [ 3.651430] debug: unmapping init [mem 0xffffffff9c982000-0xffffffff9cc58fff] [ 3.659055] Write protecting the kernel read-only data: 28672k [ 3.662282] debug: unmapping init [mem 0xffffffff9b003000-0xffffffff9b1fffff] [ 3.665718] debug: unmapping init [mem 0xffffffff9b914000-0xffffffff9b9fffff] [ 3.693777] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.700341] systemd[1]: Detected virtualization kvm. [ 3.702294] systemd[1]: Detected architecture x86-64. [ 3.703628] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.730518] systemd[1]: No hostname configured. [ 3.732038] systemd[1]: Set hostname to . [ 3.734053] random: systemd: uninitialized urandom read (16 bytes read) [ 3.736352] systemd[1]: Initializing machine ID from random generator. [ 3.775832] random: ln: uninitialized urandom read (6 bytes read) [ 3.853395] random: systemd: uninitialized urandom read (16 bytes read) [ 3.856353] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.859383] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.863295] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.651946] device-mapper: uevent: version 1.0.3 [ 4.653738] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.557111] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.616531] virtio_net virtio0 ens2: renamed from eth0 [ 5.773884] scsi host0: ata_piix [ 5.785470] scsi host1: ata_piix [ 5.788509] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.794424] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.817448] hrtimer: interrupt took 16221366 ns [ 10.152580] random: crng init done [ 10.154670] random: 7 urandom warning(s) missed due to ratelimiting [ 11.358438] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.626062] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.761567] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.865079] SELinux: Disabled at runtime. [ 18.943727] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.955658] systemd[1]: Detected virtualization kvm. [ 18.958210] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.354954] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.359291] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.365569] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.371123] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.385981] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.399838] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.452612] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ 21.533946] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 23.354693] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.730740] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.775359] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.946971] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.576851] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 30.382781] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (10s / no limit)[ 31.125288] NFS: Registering the id_resolver key type [ 31.132768] Key type id_resolver registered [ 31.138361] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Hostname Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg214-server login: [ 96.933250] libcfs: loading out-of-tree module taints kernel. [ 96.962459] Key type ._llcrypt registered [ 96.964293] Key type .llcrypt registered [ 97.036252] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 112.372640] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 113.741336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.763377] alg: No test for adler32 (adler32-zlib) [ 115.141718] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 116.015244] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 117.759215] Key type lgssc registered [ 119.018026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.066500] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.690768] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 184.354605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.383953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.683556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.699789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.770463] Lustre: lustre-MDT0000: new disk, initializing [ 185.832170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.844504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.529165] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.287027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.378472] Lustre: 6537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.419735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.428200] Lustre: Skipped 1 previous similar message [ 203.515869] Lustre: lustre-MDT0001: new disk, initializing [ 203.571990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.601703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.616031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.752927] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.900914] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 221.100180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.324752] Lustre: lustre-OST0000: new disk, initializing [ 221.327357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.344414] Lustre: 8474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.433132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.820495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.826561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.872686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.969240] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.895221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.020640] Lustre: lustre-OST0001: new disk, initializing [ 240.027988] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.036726] Lustre: 9545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.095799] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.713782] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.915555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.924715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.962662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.156901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.218188] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.706035] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.961308] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing yml_node [ 280.210326] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 283.188416] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 285.354272] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 286.622565] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 04:27:55 EDT 2026 [ 303.210102] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 304.726664] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.480318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.481096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.500820] Lustre: Skipped 1 previous similar message [ 320.526852] Lustre: Skipped 3 previous similar messages [ 325.601344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.605104] Lustre: Skipped 3 previous similar messages [ 330.723389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.731885] Lustre: Skipped 3 previous similar messages [ 334.303186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.546050] Lustre: server umount lustre-MDT0000 complete [ 337.747512] LustreError: 10361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177727 with bad export cookie 5232454655482756865 [ 337.753927] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.776743] LustreError: 10361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.960940] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.962609] LustreError: 6543:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.964373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.982323] Lustre: Skipped 3 previous similar messages [ 341.012109] LustreError: 6543:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 6 previous similar messages [ 343.008355] LustreError: 6543:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.026636] LustreError: 6543:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 348.140977] LustreError: 6544:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.141531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.165076] LustreError: 6544:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 348.193538] Lustre: Skipped 4 previous similar messages [ 352.223162] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.511974] Lustre: server umount lustre-MDT0001 complete [ 356.054317] Lustre: server umount lustre-OST0000 complete [ 359.272767] Lustre: server umount lustre-OST0001 complete [ 365.139622] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 375.377093] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 416.162588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.421309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.451995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.529273] Lustre: lustre-MDT0000: new disk, initializing [ 416.631412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.670623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.286569] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 432.922985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.023466] Lustre: 15105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.077139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.080813] Lustre: Skipped 1 previous similar message [ 433.137392] Lustre: lustre-MDT0001: new disk, initializing [ 433.229748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.267748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.290442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.015661] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 448.423642] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.166934] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.502274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.728973] Lustre: lustre-OST0000: new disk, initializing [ 462.733081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.738727] Lustre: 17070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.834905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.854506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.859653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.933617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.400506] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 479.098445] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.815023] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.090667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.025681] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 488.165639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.571486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.573019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.584638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.706430] Lustre: server umount lustre-OST0000 complete [ 504.802857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.802876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.825243] LustreError: Skipped 1 previous similar message [ 504.847810] Lustre: Skipped 1 previous similar message [ 506.802597] Lustre: server umount lustre-MDT0000 complete [ 509.920843] LustreError: 15110:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.961950] LustreError: 15110:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 510.825157] LustreError: 15908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177900 with bad export cookie 5232454655482758398 [ 510.832569] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.837604] LustreError: 15908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 511.150890] Lustre: server umount lustre-MDT0001 complete [ 517.475403] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:31:45 (1786177905) [ 537.726347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.237249] LustreError: 19944:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.354838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.223072] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 548.834363] LustreError: 19945:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.870817] LustreError: 19945:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 552.961900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.981201] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 566.741860] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 572.965412] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 575.550052] Lustre: Failing over lustre-MDT0000 [ 576.068971] Lustre: server umount lustre-MDT0000 complete [ 578.658264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.661406] LustreError: 19945:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.678051] Lustre: Skipped 2 previous similar messages [ 594.412666] Lustre: 3672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786177968/real 1786177968] req@ffff96ca3b25f100 x1872942863642624/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786177984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 594.448861] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.182140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.148752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 606.154775] Lustre: Skipped 1 previous similar message [ 606.232171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 611.288227] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 611.298272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 611.326160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 611.350612] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 622.032535] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 628.706035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.730067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.732490] Lustre: Skipped 3 previous similar messages [ 633.268216] Lustre: server umount lustre-MDT0000 complete [ 633.824199] LustreError: 19945:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.851570] LustreError: 19945:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 16 previous similar messages [ 637.281979] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178027 with bad export cookie 5232454655482759812 [ 637.307168] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.311512] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 637.606190] Lustre: server umount lustre-MDT0001 complete [ 646.684413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.282124] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 661.433397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.936413] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 674.338795] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 680.311507] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 682.370949] Lustre: Failing over lustre-MDT0000 [ 682.388155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.391946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.395036] Lustre: Skipped 2 previous similar messages [ 682.401795] Lustre: Skipped 1 previous similar message [ 682.541932] Lustre: server umount lustre-MDT0000 complete [ 700.713417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.900918] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.146657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 705.070229] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 706.212150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 706.220989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 706.230525] Lustre: Skipped 1 previous similar message [ 706.270846] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 714.117202] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 720.867673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.887940] Lustre: Skipped 1 previous similar message [ 724.477201] Lustre: server umount lustre-MDT0000 complete [ 725.984286] LustreError: 23284:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.000046] LustreError: 23284:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 10 previous similar messages [ 727.668199] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178117 with bad export cookie 5232454655482760750 [ 727.680349] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 727.702417] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.938450] Lustre: server umount lustre-MDT0001 complete [ 736.566037] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:35:24 (1786178124) [ 752.559967] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 763.535303] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 805.404613] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 815.761372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.119657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 816.189740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 816.336759] Lustre: lustre-MDT0000: new disk, initializing [ 816.588056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.592575] Lustre: Skipped 3 previous similar messages [ 816.606411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 821.739856] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 832.320986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.470225] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 832.720243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 837.463838] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 846.837646] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 852.010285] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 859.080118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.522866] Lustre: 31621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 861.412523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 861.478708] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 861.584616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 861.605317] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 864.855849] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 875.278343] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.945432] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 902.635459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 902.639134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.640715] LustreError: Skipped 1 previous similar message [ 902.661554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 902.661562] Lustre: Skipped 2 previous similar messages [ 905.363980] Lustre: server umount lustre-OST0000 complete [ 909.792333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 915.068452] Lustre: server umount lustre-MDT0000 complete [ 918.002056] LustreError: 30507:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.024877] LustreError: 30507:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 919.148570] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178309 with bad export cookie 5232454655482761156 [ 919.154944] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.160806] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 919.809476] Lustre: server umount lustre-MDT0001 complete [ 929.568612] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 932.080867] Key type lgssc unregistered [ 932.375803] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.381358] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.395888] LNet: Removed LNI 192.168.202.114@tcp [ 933.270142] Key type .llcrypt unregistered [ 933.273420] Key type ._llcrypt unregistered [ 955.463732] Key type ._llcrypt registered [ 955.471364] Key type .llcrypt registered [ 955.579305] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 956.616612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.650599] alg: No test for adler32 (adler32-zlib) [ 957.639341] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 957.832941] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 959.480225] Key type lgssc registered [ 960.267154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 967.679837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.714927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.267756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.759837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 979.431125] LustreError: 35017:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.461297] LustreError: 35017:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 983.789435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.438165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.866868] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 996.689982] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1002.967411] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1009.859260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.155643] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1010.287189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.411169] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1016.694861] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1024.362512] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.171226] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:40:27 (1786178427) [ 1041.459634] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1043.594664] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1053.823370] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:40:40 (1786178440) [ 1065.931206] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:40:53 (1786178453) [ 1067.454051] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1076.156570] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1076.165172] Lustre: Skipped 2 previous similar messages [ 1077.818970] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:41:06 (1786178466) [ 1081.777201] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1081.781915] Lustre: Skipped 3 previous similar messages [ 1101.536527] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1101.542419] Lustre: Skipped 6 previous similar messages [ 1104.944424] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:41:32 (1786178492) [ 1108.617546] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1112.169831] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1121.701789] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:41:49 (1786178509) [ 1124.375259] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1124.383114] Lustre: Skipped 3 previous similar messages [ 1139.114131] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:42:07 (1786178527) [ 1153.002292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.002933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.018213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.020748] LustreError: Skipped 1 previous similar message [ 1153.048844] Lustre: Skipped 1 previous similar message [ 1157.681893] Lustre: server umount lustre-OST0000 complete [ 1163.234161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.237489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.247722] Lustre: Skipped 1 previous similar message [ 1163.256590] Lustre: Skipped 2 previous similar messages [ 1167.620361] Lustre: server umount lustre-MDT0000 complete [ 1168.355478] LustreError: 35011:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.381984] LustreError: 35011:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1171.258501] LustreError: 38579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178561 with bad export cookie 11318125839419583213 [ 1171.267217] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.272467] LustreError: 38579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1171.572940] Lustre: server umount lustre-MDT0001 complete [ 1181.985693] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1184.503340] Key type lgssc unregistered [ 1184.806600] LNet: 42864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.816351] LNetError: 42864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.827072] LNet: Removed LNI 192.168.202.114@tcp [ 1185.541562] Key type .llcrypt unregistered [ 1185.543235] Key type ._llcrypt unregistered [ 1201.503471] Key type ._llcrypt registered [ 1201.505524] Key type .llcrypt registered [ 1201.594637] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 1219.795762] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1221.072195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.184572] alg: No test for adler32 (adler32-zlib) [ 1222.297845] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1222.709843] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1224.503437] Key type lgssc registered [ 1225.785226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.183500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.226906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.545554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1261.588801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1261.664501] Lustre: lustre-MDT0000: new disk, initializing [ 1261.777412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.825601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1266.372276] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1277.074645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.227125] Lustre: 46016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1277.287779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1277.296215] Lustre: Skipped 1 previous similar message [ 1277.368351] Lustre: lustre-MDT0001: new disk, initializing [ 1277.448885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.478287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1277.493622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1281.257259] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1289.650324] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1296.255337] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1304.393336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.991625] Lustre: lustre-OST0000: new disk, initializing [ 1305.000356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1305.009437] Lustre: 47987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1305.206770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.699258] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1312.828159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1312.833220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1312.941294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1321.372592] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1326.429556] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1326.663845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1331.981735] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1332.256289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1338.850654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.854617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.873861] Lustre: Skipped 1 previous similar message [ 1338.887200] Lustre: Skipped 1 previous similar message [ 1340.360800] Lustre: server umount lustre-OST0000 complete [ 1343.970050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1343.972112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.981782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1343.985144] Lustre: Skipped 1 previous similar message [ 1349.094250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.108573] Lustre: Skipped 1 previous similar message [ 1354.211124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.215566] Lustre: Skipped 1 previous similar message [ 1358.303640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1358.319905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.334193] Lustre: Skipped 1 previous similar message [ 1358.558194] Lustre: server umount lustre-MDT0000 complete [ 1361.968562] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178751 with bad export cookie 6043841676665970891 [ 1361.979270] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.983573] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1362.357780] Lustre: server umount lustre-MDT0001 complete [ 1372.278259] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:46:00 (1786178760) [ 1378.558130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.941463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.972934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1384.429179] LustreError: 50448:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.446063] LustreError: 50448:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1388.521784] LustreError: 50449:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.951238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.279591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1396.701244] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1403.879299] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.746264] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1415.028580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.420857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.260502] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1427.773093] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.197603] LustreError: 52976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1432.980954] Lustre: Modifying parameter general.timeout=1 in log params [ 1434.811691] Lustre: Modifying parameter general.timeout=3 in log params [ 1434.814869] Lustre: Skipped 1 previous similar message [ 1437.363683] Lustre: Modifying parameter general.timeout=6 in log params [ 1437.366189] Lustre: Skipped 2 previous similar messages [ 1441.539048] Lustre: Modifying parameter general.timeout=11 in log params [ 1441.543945] Lustre: Skipped 4 previous similar messages [ 1449.629405] Lustre: Modifying parameter general.timeout=21 in log params [ 1449.633544] Lustre: Skipped 9 previous similar messages [ 1466.304649] Lustre: Modifying parameter general.timeout=40 in log params [ 1466.308966] Lustre: Skipped 18 previous similar messages [ 1498.636426] Lustre: Modifying parameter general.timeout=76 in log params [ 1498.642092] Lustre: Skipped 35 previous similar messages [ 1556.035391] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:49:04 (1786178944) [ 1559.120769] Lustre: 59922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1561.066483] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1563.389996] Lustre: 60130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1564.509366] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1564.518512] Lustre: Skipped 58 previous similar messages [ 1568.226551] Lustre: 60326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1568.244744] Lustre: 60326:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1603.300380] Lustre: 61805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1603.307882] Lustre: 61805:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1603.328926] LustreError: 61805:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1607.428718] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:49:55 (1786178995) [ 1620.963819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.973195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.977786] Lustre: Skipped 3 previous similar messages [ 1620.996036] Lustre: Skipped 2 previous similar messages [ 1624.612840] Lustre: server umount lustre-MDT0000 complete [ 1626.088361] LustreError: 50448:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.099221] LustreError: 50448:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1627.850669] LustreError: 50430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179017 with bad export cookie 6043841676665971773 [ 1627.852135] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.864485] LustreError: 50430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1628.200695] Lustre: server umount lustre-MDT0001 complete [ 1642.192753] Lustre: server umount lustre-OST0000 complete [ 1672.506282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.856397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1673.236997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.303201] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1686.270088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.322333] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1686.377034] Lustre: 64339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1691.142818] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1698.454871] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1704.185420] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1710.923473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.157476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1711.539525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.548568] Lustre: Skipped 1 previous similar message [ 1717.441885] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1723.857695] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1725.240507] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1725.247978] Lustre: Skipped 18 previous similar messages [ 1725.300729] Lustre: 66903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1733.600870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.606632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.618948] LustreError: Skipped 1 previous similar message [ 1733.619747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.651173] Lustre: Skipped 1 previous similar message [ 1736.677341] Lustre: server umount lustre-OST0000 complete [ 1742.816246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1742.825731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.599386] Lustre: server umount lustre-MDT0000 complete [ 1748.961211] LustreError: 65796:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.988241] LustreError: 65796:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 1750.075372] LustreError: 64331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179139 with bad export cookie 6043841676665974846 [ 1750.082331] LustreError: 64331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1750.083291] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.508262] Lustre: server umount lustre-MDT0001 complete [ 1761.035572] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1763.791526] Key type lgssc unregistered [ 1764.027699] LNet: 68116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.034181] LNetError: 68116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.059033] LNet: Removed LNI 192.168.202.114@tcp [ 1764.892206] Key type .llcrypt unregistered [ 1764.894403] Key type ._llcrypt unregistered [ 1786.276944] Key type ._llcrypt registered [ 1786.279852] Key type .llcrypt registered [ 1786.407815] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1787.412216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.466784] alg: No test for adler32 (adler32-zlib) [ 1788.579967] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1788.812599] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1790.463178] Key type lgssc registered [ 1791.645886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1798.878590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1798.898241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.444162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.656683] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1810.934899] LustreError: 69533:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.958611] LustreError: 69533:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1814.389564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.207700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.469579] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1826.180567] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1831.692741] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1838.089557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.754453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.368685] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1852.527353] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.637343] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1871.328574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.347611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.393762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.854262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.865415] Lustre: Skipped 1 previous similar message [ 1874.518405] Lustre: server umount lustre-MDT0000 complete [ 1876.961251] LustreError: 69532:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.987698] LustreError: 69532:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 1877.791256] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179267 with bad export cookie 17885721209393964832 [ 1877.804667] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.808851] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1878.096538] Lustre: server umount lustre-MDT0001 complete [ 1892.070375] Lustre: server umount lustre-OST0000 complete [ 1917.383786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.772753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1917.830080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1918.092661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.300743] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1930.354543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.512042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1930.533445] Lustre: 74402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.000293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.647663] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1942.713959] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1947.580766] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1953.981702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1954.167662] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1954.535039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.620550] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1967.626396] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.189195] Lustre: 76974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1970.984874] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1976.806461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1976.811231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.836057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.849186] Lustre: Skipped 2 previous similar messages [ 1981.874901] Lustre: server umount lustre-OST0000 complete [ 1987.046727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1987.052192] LustreError: Skipped 1 previous similar message [ 1987.057991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.065426] Lustre: Skipped 1 previous similar message [ 1987.070086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1987.073459] Lustre: Skipped 2 previous similar messages [ 1992.689463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.692055] Lustre: Skipped 1 previous similar message [ 1997.797492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.802342] Lustre: Skipped 2 previous similar messages [ 1999.327535] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1999.573254] Lustre: server umount lustre-MDT0000 complete [ 2002.915947] LustreError: 74413:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.950038] LustreError: 74413:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2003.846481] LustreError: 74394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179393 with bad export cookie 17885721209393965672 [ 2003.856787] LustreError: 74394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2003.859116] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.306247] Lustre: server umount lustre-MDT0001 complete [ 2017.868310] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2021.151082] Key type lgssc unregistered [ 2021.424290] LNet: 78189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.433911] LNetError: 78189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.450708] LNet: Removed LNI 192.168.202.114@tcp [ 2022.441167] Key type .llcrypt unregistered [ 2022.442935] Key type ._llcrypt unregistered [ 2047.899074] Key type ._llcrypt registered [ 2047.901959] Key type .llcrypt registered [ 2048.001617] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2049.507070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.566143] alg: No test for adler32 (adler32-zlib) [ 2050.588268] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2050.839795] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2052.495185] Key type lgssc registered [ 2053.965591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.759514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2061.773871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.351972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.349433] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2073.575300] LustreError: 79602:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.598559] LustreError: 79602:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2077.077944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.830734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.428495] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2089.528835] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.950800] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.207708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.623161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.012995] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2114.365649] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2121.411723] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2122.442768] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2124.584910] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2124.587400] Lustre: Skipped 1 previous similar message [ 2127.170269] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2127.174340] Lustre: Skipped 2 previous similar messages [ 2131.642803] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2131.646972] Lustre: Skipped 4 previous similar messages [ 2139.809438] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2139.813070] Lustre: Skipped 8 previous similar messages [ 2156.678902] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2156.683274] Lustre: Skipped 19 previous similar messages [ 2189.276230] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2189.280253] Lustre: Skipped 36 previous similar messages [ 2217.269362] Lustre: 87703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2219.782775] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:00:08 (1786179608) [ 2221.384938] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2223.324416] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:00:11 (1786179611) [ 2230.752783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2230.757049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.769232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.779902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.781341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.422440] Lustre: server umount lustre-OST0000 complete [ 2236.916728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.939641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.947348] Lustre: Skipped 1 previous similar message [ 2241.521677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2241.532109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.551571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.137721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.146415] Lustre: Skipped 3 previous similar messages [ 2251.232849] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2251.442031] Lustre: server umount lustre-MDT0000 complete [ 2252.257394] LustreError: 79597:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.297665] LustreError: 79597:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2255.301292] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179645 with bad export cookie 11669728064592557553 [ 2255.303812] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.314840] LustreError: 79582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2255.675576] Lustre: server umount lustre-MDT0001 complete [ 2267.446170] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2271.255688] Key type lgssc unregistered [ 2271.919342] LNet: 88961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.944794] LNetError: 88961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.983869] LNet: Removed LNI 192.168.202.114@tcp [ 2273.447517] Key type .llcrypt unregistered [ 2273.450880] Key type ._llcrypt unregistered [ 2279.041253] Key type ._llcrypt registered [ 2279.045478] Key type .llcrypt registered [ 2279.228771] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2283.331487] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules [ 2284.155742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.267831] alg: No test for adler32 (adler32-zlib) [ 2285.330848] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2285.546658] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2287.271592] Key type lgssc registered [ 2288.222338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.363890] LustreError: 90172:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2305.134171] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2311.143912] LustreError: 90172:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2313.138692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.153204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.552423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.817130] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2324.963269] LustreError: 91196:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.994435] LustreError: 91196:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2327.860583] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:01:56 (1786179716) [ 2330.086688] LustreError: 91197:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.599240] LustreError: 91228:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2332.607944] LustreError: 91228:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2332.618147] LustreError: 91228:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2332.939718] Lustre: server umount lustre-MDT0000 complete [ 2345.383517] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2348.124979] Key type lgssc unregistered [ 2348.397822] LNet: 92723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.400970] LNetError: 92723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.412737] LNet: Removed LNI 192.168.202.114@tcp [ 2349.283552] Key type .llcrypt unregistered [ 2349.285898] Key type ._llcrypt unregistered [ 2369.616925] Key type ._llcrypt registered [ 2369.619387] Key type .llcrypt registered [ 2369.753484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2370.805608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.851668] alg: No test for adler32 (adler32-zlib) [ 2372.549092] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2372.881465] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2374.571223] Key type lgssc registered [ 2376.271258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.309141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.325823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.951429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.366515] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2397.160209] LustreError: 93948:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.217890] LustreError: 93948:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2400.496790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.808238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.354183] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2412.017755] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2417.633035] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2423.972188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.495185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.114047] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2437.800333] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.992322] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:04:29 (1786179869) [ 2483.871762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2485.831307] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:04:33 (1786179873) [ 2489.313412] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.320857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2489.344047] Lustre: Skipped 2 previous similar messages [ 2489.361610] Lustre: Skipped 2 previous similar messages [ 2494.433955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.450086] Lustre: Skipped 2 previous similar messages [ 2495.573525] Lustre: server umount lustre-MDT0000 complete [ 2499.207648] LustreError: 93930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179889 with bad export cookie 1011807329013538817 [ 2499.211118] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.216020] LustreError: 93930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2499.480905] Lustre: server umount lustre-MDT0001 complete [ 2513.927232] Lustre: server umount lustre-OST0000 complete [ 2526.732091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.150420] LustreError: 98220:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.380645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.118721] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2532.833448] LustreError: 98221:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.954767] LustreError: 98220:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.160987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.997544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.238294] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2551.627923] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2556.357862] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2568.460462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.654656] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2568.659507] LustreError: 98210:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2568.678334] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2568.684431] LustreError: 100303:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2568.688978] LustreError: 100303:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2568.694511] LustreError: 100303:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2568.738419] Lustre: server umount lustre-OST0000 complete [ 2568.741225] LustreError: 100303:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2576.543062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.668656] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2576.677045] LustreError: 98208:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2576.692926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2576.697961] LustreError: 100770:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2576.702586] LustreError: 100770:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2576.705894] LustreError: 100770:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2576.778537] Lustre: server umount lustre-OST0000 complete [ 2576.785987] LustreError: 100770:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2585.163770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.371463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2585.375981] Lustre: Found index 0 for lustre-OST0000, updating log [ 2585.382464] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2585.467531] Lustre: lustre-OST0000: new disk, initializing [ 2585.472400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2585.482827] Lustre: 101306:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2585.726095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.627235] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2594.724079] Lustre: Failing over lustre-OST0000 [ 2594.822621] Lustre: server umount lustre-OST0000 complete [ 2601.952678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.961159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.973039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.978328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.366483] Lustre: server umount lustre-MDT0000 complete [ 2608.100844] LustreError: 98220:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.115621] LustreError: 98220:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 2608.554230] LustreError: 98200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179998 with bad export cookie 1011807329013541561 [ 2608.560080] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.562974] LustreError: 98200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2619.034820] Lustre: server umount lustre-MDT0001 complete [ 2627.554684] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:06:55 (1786180015) [ 2643.579922] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2653.019300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.271697] LustreError: 103650:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.386813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.534935] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2664.877099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.277373] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2671.830336] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2678.002117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.494292] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2679.528919] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2679.543399] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2679.575361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2684.510678] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2684.919098] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2684.936055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2690.031984] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 2690.050212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2691.062762] Lustre: lustre-OST0000: Recovery over after 0:12, of 1 clients 1 recovered and 0 were evicted. [ 2691.063148] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2694.036566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.149025] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2694.236480] Lustre: lustre-OST0001: new disk, initializing [ 2694.240903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2694.248407] Lustre: 105972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2694.576578] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.589898] Lustre: Skipped 2 previous similar messages [ 2700.871348] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2704.010367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2704.041563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2704.168874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2708.519601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2717.437586] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2721.761463] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.768521] LustreError: Skipped 1 previous similar message [ 2721.772505] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.786822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2721.795490] Lustre: Skipped 2 previous similar messages [ 2727.435238] Lustre: server umount lustre-MDT0001 complete [ 2729.108199] LustreError: 105531:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.132736] LustreError: 105531:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 2738.072092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.303887] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2738.317085] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2738.412372] Lustre: 103637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.511785] LustreError: 107477:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2739.529514] LustreError: 107477:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2739.549920] LustreError: 103643:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2739.761486] LustreError: 107493:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -17 [ 2739.799068] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2739.799068] [ 2744.848156] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2745.574896] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2755.910625] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:09:03 (1786180143) [ 2774.671603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2792.242783] Lustre: 109586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2814.996230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2818.787334] Lustre: 110721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2897.892673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2897.910841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.930678] Lustre: Skipped 2 previous similar messages [ 2897.948313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2897.959549] Lustre: Skipped 4 previous similar messages [ 2904.260902] Lustre: server umount lustre-MDT0000 complete [ 2908.673627] LustreError: 103647:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.727263] LustreError: 103647:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 7 previous similar messages [ 2908.799093] LustreError: 104394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180298 with bad export cookie 1011807329013542562 [ 2908.803385] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2908.827895] LustreError: 104394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2909.295809] Lustre: server umount lustre-MDT0001 complete [ 2923.414806] Lustre: server umount lustre-OST0000 complete [ 2936.718849] Lustre: server umount lustre-OST0001 complete [ 2942.092718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2951.355121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2960.478151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2972.628904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2996.355497] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3006.583234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.599185] Lustre: lustre-MDT0000: reset Object Index mappings [ 3006.974139] LustreError: 114670:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.985385] LustreError: 114670:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 3007.087093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.095193] Lustre: Skipped 1 previous similar message [ 3011.287726] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3019.822642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.843255] Lustre: lustre-MDT0001: reset Object Index mappings [ 3024.154186] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3026.631622] Lustre: 115821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3033.014340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.946921] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3045.176399] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3048.860313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.663468] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3051.669237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3051.679487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3057.400304] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3064.885427] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3069.201673] Lustre: 117699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3116.501090] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:15:04 (1786180504) [ 3144.167607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.176407] Lustre: Skipped 4 previous similar messages [ 3144.181874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.198312] Lustre: Skipped 11 previous similar messages [ 3150.149383] Lustre: server umount lustre-MDT0000 complete [ 3154.297453] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180544 with bad export cookie 1011807329013589455 [ 3154.301985] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3154.309318] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3154.405545] LustreError: 118057:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.441906] LustreError: 118057:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 5 previous similar messages [ 3160.863952] Lustre: server umount lustre-MDT0001 complete [ 3170.760805] Lustre: server umount lustre-OST0000 complete [ 3188.191193] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3188.603033] Lustre: server umount lustre-OST0001 complete [ 3194.954570] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 3204.230893] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3250.841361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.071034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3251.096204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3251.161908] Lustre: lustre-MDT0000: new disk, initializing [ 3251.244628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.249858] Lustre: Skipped 3 previous similar messages [ 3251.260955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3255.667232] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3262.651350] Lustre: server umount lustre-MDT0000 complete [ 3271.198285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.403786] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3271.414845] Lustre: 121972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3271.424825] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3276.681948] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3281.691657] Lustre: server umount lustre-MDT0000 complete [ 3294.001352] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:18:00 (1786180680) [ 3296.603394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3298.394598] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:18:06 (1786180686) [ 3317.927816] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3330.807592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.935757] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3347.620212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.751743] Lustre: 124022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3347.800140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3347.807612] Lustre: Skipped 1 previous similar message [ 3347.892515] Lustre: lustre-MDT0001: new disk, initializing [ 3348.008678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3348.025844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3352.203393] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3356.960644] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3364.622756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.151423] Lustre: lustre-OST0000: new disk, initializing [ 3365.157477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3365.171698] Lustre: 125604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3366.330969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3366.346337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3366.511548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3371.108761] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3381.985622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3382.148499] Lustre: lustre-OST0001: new disk, initializing [ 3382.158838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3382.164617] Lustre: 126472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3383.550910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3383.560203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3383.755114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3389.275176] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3399.767196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3403.607927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3781.201789] Lustre: 132883:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3781.219364] Lustre: 132883:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3781.249070] Lustre: 132883:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3781.264839] Lustre: 132883:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3781.278466] Lustre: 132883:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3781.299538] Lustre: 132883:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4058.598457] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:30:46 (1786181446) [ 4074.977747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4074.983445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.985847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.985855] Lustre: Skipped 8 previous similar messages [ 4075.057578] Lustre: Skipped 7 previous similar messages [ 4080.102616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4080.119249] Lustre: Skipped 6 previous similar messages [ 4080.900959] Lustre: server umount lustre-MDT0000 complete [ 4084.518960] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786181474 with bad export cookie 1011807329013656564 [ 4084.523049] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.530420] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4085.218934] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.221654] LustreError: 127276:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.223187] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4085.252881] Lustre: Skipped 1 previous similar message [ 4085.282958] LustreError: 127276:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 4091.171886] Lustre: server umount lustre-MDT0001 complete [ 4109.279173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.451659] Lustre: server umount lustre-OST0000 complete [ 4127.199180] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4127.562813] Lustre: server umount lustre-OST0001 complete [ 4134.724186] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4146.721806] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4187.712941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.088291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4188.123142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4188.221996] Lustre: lustre-MDT0000: new disk, initializing [ 4188.281612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4188.288096] Lustre: Skipped 5 previous similar messages [ 4188.299970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4192.825856] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4204.395196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.654108] Lustre: 135935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4204.784164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4204.789283] Lustre: Skipped 1 previous similar message [ 4204.888700] Lustre: lustre-MDT0001: new disk, initializing [ 4205.094359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4205.112271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4210.033973] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4219.835636] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4226.595291] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4233.462477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.684975] Lustre: lustre-OST0000: new disk, initializing [ 4233.687697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4233.691725] Lustre: 137904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4233.770765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.780223] Lustre: Skipped 1 previous similar message [ 4235.294909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4235.303803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4235.438835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4240.520501] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4250.560813] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4260.393895] Lustre: lustre-MDD0000: changelog on [ 4263.353858] Lustre: lustre-MDD0001: changelog on [ 5978.858934] Lustre: lustre-MDD0001: changelog off [ 5980.574800] Lustre: 139368:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5981.435207] Lustre: lustre-MDD0000: changelog off [ 5989.352434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.361753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.374183] Lustre: Skipped 1 previous similar message [ 5989.387377] Lustre: Skipped 4 previous similar messages [ 5993.873795] Lustre: server umount lustre-OST0000 complete [ 5999.586062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.591052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.604379] Lustre: Skipped 1 previous similar message [ 5999.616468] Lustre: Skipped 1 previous similar message [ 6004.704839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.712032] Lustre: Skipped 1 previous similar message [ 6009.825524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.838135] Lustre: Skipped 1 previous similar message [ 6012.383210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6012.795627] Lustre: server umount lustre-MDT0000 complete [ 6014.945788] LustreError: 135948:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.975571] LustreError: 135948:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 6016.488689] LustreError: 135929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183406 with bad export cookie 1011807329014479785 [ 6016.489982] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6016.511362] LustreError: 135929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6016.790520] Lustre: server umount lustre-MDT0001 complete [ 6027.556156] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 6030.885595] Key type lgssc unregistered [ 6031.164662] LNet: 147043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6031.175622] LNetError: 147043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6031.190906] LNet: Removed LNI 192.168.202.114@tcp [ 6032.007292] Key type .llcrypt unregistered [ 6032.011614] Key type ._llcrypt unregistered [ 6055.459796] Key type ._llcrypt registered [ 6055.463794] Key type .llcrypt registered [ 6055.590469] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6073.266384] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6074.781829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6074.931488] alg: No test for adler32 (adler32-zlib) [ 6076.064508] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 6076.279637] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6077.959175] Key type lgssc registered [ 6079.318413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6114.755461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6114.775764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.199192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6116.264627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6116.451563] Lustre: lustre-MDT0000: new disk, initializing [ 6116.629504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6116.644643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6122.859336] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6137.549583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.770877] Lustre: 150247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6137.924565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6137.930573] Lustre: Skipped 1 previous similar message [ 6138.048557] Lustre: lustre-MDT0001: new disk, initializing [ 6138.201589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6138.258066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6138.270200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6142.889259] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6154.238788] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6160.387174] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6168.876924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6169.264516] Lustre: lustre-OST0000: new disk, initializing [ 6169.271935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6169.280577] Lustre: 152220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6169.449607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.935193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6175.966436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6176.162659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6178.220347] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6190.859449] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6199.244932] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_module obdecho/obdecho [ 6201.555898] LustreError: 153226:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6201.567737] LustreError: 153226:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6203.820361] LustreError: 153323:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6220.258309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6220.274484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.300414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6222.317265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.318101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6222.358578] Lustre: Skipped 2 previous similar messages [ 6223.719748] Lustre: server umount lustre-MDT0000 complete [ 6227.671920] LustreError: 150239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183617 with bad export cookie 14524469786252466253 [ 6227.702793] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.994858] Lustre: server umount lustre-MDT0001 complete [ 6242.323399] Lustre: server umount lustre-OST0000 complete [ 6263.655664] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6273.361094] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6312.233025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.559661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6312.605503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6312.672495] Lustre: lustre-MDT0000: new disk, initializing [ 6312.800868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.831711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6317.631979] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6328.417378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.523638] Lustre: 157036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6328.576358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6328.580117] Lustre: Skipped 1 previous similar message [ 6328.655517] Lustre: lustre-MDT0001: new disk, initializing [ 6328.669327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6328.733985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.773802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6328.788839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6333.299372] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6342.476788] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6348.223606] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6355.592638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6355.910767] Lustre: lustre-OST0000: new disk, initializing [ 6355.916943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6355.926545] Lustre: 159005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6356.049023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.180172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6361.200086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6361.331138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6362.372794] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6371.960627] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6376.477593] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6376.766631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6381.320337] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6381.551565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6387.175289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6387.185880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.239433] Lustre: Skipped 1 previous similar message [ 6387.249599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.269295] Lustre: Skipped 1 previous similar message [ 6389.954863] Lustre: server umount lustre-OST0000 complete [ 6395.360513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6395.367560] LustreError: Skipped 1 previous similar message [ 6395.374149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.381571] Lustre: Skipped 1 previous similar message [ 6395.387967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6400.200990] Lustre: server umount lustre-MDT0000 complete [ 6402.527845] LustreError: 158128:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.550151] LustreError: 158128:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 6404.471175] LustreError: 157031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183794 with bad export cookie 14524469786252467212 [ 6404.473423] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6404.481866] LustreError: 157031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6404.873501] Lustre: server umount lustre-MDT0001 complete [ 6408.666904] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:09:56 (1786183796) [ 6410.393704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6412.342060] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:10:00 (1786183800) [ 6421.238578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.716660] LustreError: 161419:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.780538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.252793] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6427.112544] LustreError: 161420:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.206297] LustreError: 161419:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.133493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6439.281663] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6447.417027] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6453.543813] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6461.109726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6461.594316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.600684] Lustre: Skipped 1 previous similar message [ 6468.298627] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6475.849254] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6486.501546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.524511] Lustre: Skipped 1 previous similar message [ 6486.536849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6486.537554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.549447] Lustre: Skipped 4 previous similar messages [ 6488.659152] Lustre: server umount lustre-MDT0000 complete [ 6491.633480] LustreError: 161420:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.671863] LustreError: 161420:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 6492.672725] LustreError: 161398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786183882 with bad export cookie 14524469786252468087 [ 6492.687630] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.691352] LustreError: 161398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6496.737727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.756793] Lustre: Skipped 2 previous similar messages [ 6499.087572] Lustre: server umount lustre-MDT0001 complete [ 6505.457984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6520.002496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.212451] LustreError: 165262:0:(llog.c:1655:llog_backup()) MGC192.168.202.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6520.218157] Lustre: 165262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6522.336491] LustreError: 165267:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.373650] LustreError: 165267:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 6538.875745] LustreError: 165268:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6538.902096] LustreError: 165268:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 12 previous similar messages [ 6573.537616] LustreError: 165269:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.563988] LustreError: 165269:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 25 previous similar messages [ 6581.215277] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6581.227865] Lustre: 165262:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6622.210328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6627.315116] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6627.433662] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6635.556381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.539631] LustreError: 163253:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.582256] LustreError: 163253:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 42 previous similar messages [ 6642.987133] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6643.181907] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6651.069850] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6652.584902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6658.232207] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.027706] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6679.025954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.047897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6679.054279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.054288] Lustre: Skipped 4 previous similar messages [ 6679.058063] Lustre: Skipped 1 previous similar message [ 6683.956197] Lustre: server umount lustre-MDT0000 complete [ 6688.135174] LustreError: 161398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184077 with bad export cookie 14524469786252474548 [ 6688.143970] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.145876] LustreError: 161398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6688.653237] Lustre: server umount lustre-MDT0001 complete [ 6703.032933] Lustre: server umount lustre-OST0000 complete [ 6724.114135] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6734.042319] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6771.473457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.777718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6771.816840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6771.895759] Lustre: lustre-MDT0000: new disk, initializing [ 6771.980152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6771.991593] Lustre: Skipped 1 previous similar message [ 6772.010329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6777.229372] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6788.020353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.149977] Lustre: 170364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.329730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6793.313191] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6803.243465] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6808.569337] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6817.109741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.463441] Lustre: lustre-OST0000: new disk, initializing [ 6817.466255] Lustre: Skipped 1 previous similar message [ 6817.470242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6817.476730] Lustre: Skipped 2 previous similar messages [ 6817.482356] Lustre: 172334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6822.980754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6822.997409] Lustre: Skipped 1 previous similar message [ 6823.006343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6823.163042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6824.529967] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6835.391709] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.507571] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6841.860828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6848.016049] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6848.400413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6854.116892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6854.119067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.126108] LustreError: Skipped 1 previous similar message [ 6854.135735] Lustre: Skipped 1 previous similar message [ 6854.137666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6854.149317] Lustre: Skipped 3 previous similar messages [ 6857.168198] Lustre: server umount lustre-OST0000 complete [ 6866.882502] Lustre: server umount lustre-MDT0000 complete [ 6869.477538] LustreError: 170376:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.515448] LustreError: 170376:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [ 6873.102984] LustreError: 170357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184262 with bad export cookie 14524469786252491376 [ 6873.110927] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.128038] LustreError: 170357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6873.915617] Lustre: server umount lustre-MDT0001 complete [ 6886.300287] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:17:54 (1786184274) [ 6894.392919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.800700] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6910.843508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.646756] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6924.556107] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6930.867210] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.909636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.412829] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6953.117148] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.432587] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:19:13 (1786184353) [ 6970.337300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.343413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.355304] Lustre: Skipped 3 previous similar messages [ 6970.361326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.367644] Lustre: Skipped 2 previous similar messages [ 6975.806049] Lustre: server umount lustre-OST0000 complete [ 6993.375290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.554468] Lustre: server umount lustre-MDT0000 complete [ 6996.948703] LustreError: 174728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184386 with bad export cookie 14524469786252492258 [ 6996.952062] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.969252] LustreError: 174728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6997.441217] Lustre: server umount lustre-MDT0001 complete [ 7007.948390] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7010.619594] Key type lgssc unregistered [ 7010.868774] LNet: 178483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7010.881219] LNetError: 178483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7010.907298] LNet: Removed LNI 192.168.202.114@tcp [ 7011.831845] Key type .llcrypt unregistered [ 7011.834108] Key type ._llcrypt unregistered [ 7032.784812] Key type ._llcrypt registered [ 7032.793466] Key type .llcrypt registered [ 7032.918767] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7034.373872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7034.471707] alg: No test for adler32 (adler32-zlib) [ 7035.680757] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7035.966082] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7037.687303] Key type lgssc registered [ 7039.314475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7049.356804] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7049.383017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.927421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7070.047609] LustreError: 179726:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [ 7075.167570] LustreError: 179726:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7075.187060] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7075.193324] LustreError: MGC192.168.202.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7075.209551] LustreError: 179726:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7075.222311] LustreError: 179726:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7075.229301] LustreError: 179726:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7075.320096] Lustre: server umount lustre-OST0000 complete [ 7075.323046] LustreError: 179726:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7083.905860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.456392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.041360] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7095.778149] LustreError: 180228:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.792655] LustreError: 180228:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7098.920042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.183251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.453638] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7110.683624] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7117.220446] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7125.963264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.197833] LustreError: 182044:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7126.206436] Lustre: 182044:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7126.354200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.538111] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7141.544064] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.225412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.240854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.242545] Lustre: Skipped 2 previous similar messages [ 7144.280835] Lustre: Skipped 2 previous similar messages [ 7149.438782] Lustre: server umount lustre-MDT0000 complete [ 7149.538633] LustreError: 181263:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.552832] LustreError: 181263:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7152.714265] LustreError: 180209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184542 with bad export cookie 4400215562231526479 [ 7152.733440] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.696855] LustreError: 180956:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ab251702 x1872950119656960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7153.910781] Lustre: server umount lustre-MDT0001 complete [ 7168.272984] Lustre: server umount lustre-OST0000 complete [ 7178.383350] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7189.490721] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7227.926818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.155824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7228.176548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7228.289198] Lustre: lustre-MDT0000: new disk, initializing [ 7228.407443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7228.451220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7233.124319] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7243.319292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.455509] Lustre: 185567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.541413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7243.545508] Lustre: Skipped 1 previous similar message [ 7243.645477] Lustre: lustre-MDT0001: new disk, initializing [ 7243.763593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7243.802167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7243.807686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7248.105222] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7257.076688] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7262.903807] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.788304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7271.054290] Lustre: lustre-OST0000: new disk, initializing [ 7271.057540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7271.064963] Lustre: 187534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7271.144849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7277.230498] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7280.715750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7280.736650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7280.871848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7286.451311] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7291.447788] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7291.654768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7296.456635] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7296.636523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.614866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.617508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.619281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.623052] LustreError: Skipped 1 previous similar message [ 7301.647028] Lustre: Skipped 1 previous similar message [ 7304.923962] Lustre: server umount lustre-OST0000 complete [ 7310.305127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7310.314574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.344634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.353515] Lustre: Skipped 1 previous similar message [ 7316.967171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7316.980897] Lustre: Skipped 3 previous similar messages [ 7322.083509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7322.094736] Lustre: Skipped 1 previous similar message [ 7323.103751] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7323.357883] Lustre: server umount lustre-MDT0000 complete [ 7327.205897] LustreError: 185579:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.242104] LustreError: 185579:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 7327.260704] LustreError: 185560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184717 with bad export cookie 4400215562231527200 [ 7327.270446] LustreError: 185560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7327.271114] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7327.585369] Lustre: server umount lustre-MDT0001 complete [ 7337.051620] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:25:25 (1786184725) [ 7355.366036] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7357.966366] Key type lgssc unregistered [ 7358.229272] LNet: 190484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7358.242789] LNetError: 190484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7358.266324] LNet: Removed LNI 192.168.202.114@tcp [ 7359.082615] Key type .llcrypt unregistered [ 7359.085350] Key type ._llcrypt unregistered [ 7376.214766] Key type ._llcrypt registered [ 7376.217261] Key type .llcrypt registered [ 7376.319207] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7377.256943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.338339] alg: No test for adler32 (adler32-zlib) [ 7378.566786] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 7378.756771] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7380.439200] Key type lgssc registered [ 7381.473540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7388.772915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7388.787595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.371788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.251755] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7400.931678] LustreError: 191702:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7400.962304] LustreError: 191702:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7405.277559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.856276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7411.175524] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7418.740630] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7424.602303] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7432.136721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7432.406851] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7432.410430] LustreError: 193565:0:(llog.c:1696:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7432.417678] Lustre: 193565:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7432.573264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.959393] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7446.045818] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7451.105487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.111738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7451.147439] Lustre: Skipped 2 previous similar messages [ 7451.162964] Lustre: Skipped 2 previous similar messages [ 7452.194842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7452.211651] Lustre: Skipped 2 previous similar messages [ 7454.773716] Lustre: server umount lustre-MDT0000 complete [ 7457.266966] LustreError: 191698:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7457.295646] LustreError: 191698:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7459.018383] LustreError: 191681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786184848 with bad export cookie 15087238117404464223 [ 7459.022267] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7459.027580] LustreError: 191681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7459.328458] LustreError: 192432:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000000a8db7e x1872950479318016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7459.498133] Lustre: server umount lustre-MDT0001 complete [ 7473.911621] Lustre: server umount lustre-OST0000 complete [ 7484.453490] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7495.017634] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7535.401546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.707147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7535.732786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7535.902182] Lustre: lustre-MDT0000: new disk, initializing [ 7536.006251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.055877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7541.579705] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7553.880704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.976759] Lustre: 197094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7554.010238] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7554.013800] Lustre: Skipped 1 previous similar message [ 7554.016776] LustreError: 197875:0:(llog.c:1696:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7554.022339] LustreError: 197875:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7554.026568] Lustre: 197875:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7554.041743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7554.044889] Lustre: Skipped 1 previous similar message [ 7554.124198] Lustre: lustre-MDT0001: new disk, initializing [ 7554.245961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7554.320048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7554.331830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7559.249191] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7568.949093] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7574.681268] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7581.606964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.974816] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7581.980879] Lustre: Skipped 1 previous similar message [ 7581.986687] LustreError: 199040:0:(llog.c:1696:llog_backup()) MGC192.168.202.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7582.006767] LustreError: 199040:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7582.016798] Lustre: 199040:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7582.033556] Lustre: 199040:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7582.074744] Lustre: lustre-OST0000: new disk, initializing [ 7582.083610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7582.099129] Lustre: 199063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7582.263363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.002518] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7590.493603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.506355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7590.627920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7598.258590] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7602.691931] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7602.929567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7606.935628] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7607.165540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7609.316592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.318357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.343625] Lustre: Skipped 1 previous similar message [ 7609.361743] Lustre: Skipped 1 previous similar message [ 7614.434258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7614.446627] Lustre: Skipped 1 previous similar message [ 7615.416127] Lustre: server umount lustre-OST0000 complete [ 7619.552775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.564488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7619.570429] Lustre: Skipped 1 previous similar message [ 7629.796884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.802026] Lustre: Skipped 3 previous similar messages [ 7633.375123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7633.558118] Lustre: server umount lustre-MDT0000 complete [ 7634.912223] LustreError: 197105:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.927293] LustreError: 197105:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 7637.451568] LustreError: 197086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185027 with bad export cookie 15087238117404464944 [ 7637.457048] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7637.461036] LustreError: 197086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7637.860673] Lustre: server umount lustre-MDT0001 complete [ 7647.891737] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:30:35 (1786185035) [ 7665.220388] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7674.987952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.304665] LustreError: 202192:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.367707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.242873] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7680.486966] LustreError: 202193:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.195946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.162264] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7694.026611] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7699.996134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.295611] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7714.179563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.362925] Lustre: lustre-OST0001: new disk, initializing [ 7714.367523] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7714.371923] Lustre: 204500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7714.493129] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7714.500730] Lustre: Skipped 2 previous similar messages [ 7719.951896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7719.965377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7720.048379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7720.450396] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7731.256808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7739.895695] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7741.696304] Lustre: DEBUG MARKER: ADD OST3 [ 7747.711489] loop: module loaded [ 7747.826141] loop0: detected capacity change from 0 to 204800000 [ 7748.063685] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7748.108292] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7748.132303] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7754.231643] loop0: detected capacity change from 0 to 204800000 [ 7754.891682] Lustre: DEBUG MARKER: STOP OST3 [ 7759.607533] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.838975] Lustre: lustre-OST0003: new disk, initializing [ 7759.842947] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7759.848499] Lustre: 206514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7761.250420] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7761.343476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7761.347223] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7761.377617] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7762.820127] Lustre: Failing over lustre-OST0003 [ 7762.893637] Lustre: server umount lustre-OST0003 complete [ 7763.300538] LustreError: 206669:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7765.943882] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7766.515116] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7766.527174] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.540957] LustreError: 205076:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.543790] Lustre: Skipped 2 previous similar messages [ 7766.561984] LustreError: 205076:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [ 7773.038556] Lustre: DEBUG MARKER: START OST3 again [ 7778.554027] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7778.804541] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7778.817884] Lustre: Skipped 1 previous similar message [ 7778.826149] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7780.071884] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7780.133446] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7780.134481] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7784.077412] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7806.597717] Lustre: Failing over lustre-OST0003 [ 7806.680260] Lustre: server umount lustre-OST0003 complete [ 7809.517504] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7809.527095] LustreError: 205076:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7809.534520] LustreError: 205076:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 8 previous similar messages [ 7829.985629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7829.994331] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7830.009928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.035450] Lustre: Skipped 3 previous similar messages [ 7830.053395] Lustre: Skipped 3 previous similar messages [ 7833.575152] Lustre: server umount lustre-MDT0000 complete [ 7836.968383] LustreError: 202174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185226 with bad export cookie 15087238117404465826 [ 7836.989616] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.995404] LustreError: 202174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7843.622346] Lustre: server umount lustre-MDT0001 complete [ 7847.214787] Lustre: server umount lustre-OST0000 complete [ 7850.320377] Lustre: server umount lustre-OST0001 complete [ 7856.597355] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7864.244788] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7903.750712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.912427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7903.961159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7904.032411] Lustre: lustre-MDT0000: new disk, initializing [ 7904.118389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7908.899415] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7919.610502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.791467] Lustre: 211957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7920.133050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.140630] Lustre: Skipped 1 previous similar message [ 7920.209399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7920.230856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7925.081299] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7934.017792] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.197958] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7948.419797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.856906] Lustre: lustre-OST0000: new disk, initializing [ 7948.864815] Lustre: Skipped 1 previous similar message [ 7948.875831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7948.885645] Lustre: Skipped 2 previous similar messages [ 7948.906437] Lustre: 213929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7950.669408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7950.821923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7956.670962] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7967.385768] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.933891] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7973.250698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7978.635343] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7979.025270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7981.547092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7981.553744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.568860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7981.581736] Lustre: Skipped 4 previous similar messages [ 7981.595876] Lustre: Skipped 5 previous similar messages [ 7995.359229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7995.506589] Lustre: server umount lustre-OST0000 complete [ 8005.629187] Lustre: server umount lustre-MDT0000 complete [ 8007.138230] LustreError: 211970:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.157508] LustreError: 211970:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 17 previous similar messages [ 8009.286344] LustreError: 212763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185399 with bad export cookie 15087238117404468122 [ 8009.288086] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8009.308258] LustreError: 212763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8009.709607] Lustre: server umount lustre-MDT0001 complete [ 8013.899152] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:36:41 (1786185401) [ 8031.656509] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8042.770403] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8080.077857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.472813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8080.502206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8080.658779] Lustre: lustre-MDT0000: new disk, initializing [ 8080.751928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8080.757607] Lustre: Skipped 1 previous similar message [ 8085.442309] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8095.717877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8095.954897] Lustre: 218540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8096.210790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8100.224455] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8110.541452] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8115.974535] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8122.187567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8122.503433] Lustre: 220512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8123.943909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8128.358504] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8137.643431] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8142.573786] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8142.718402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8147.993785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8148.285410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8154.596201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.597461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.614945] Lustre: Skipped 2 previous similar messages [ 8154.626672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8154.626681] Lustre: Skipped 6 previous similar messages [ 8156.885202] Lustre: server umount lustre-OST0000 complete [ 8162.785082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8175.074140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8175.461717] Lustre: server umount lustre-MDT0000 complete [ 8179.460291] LustreError: 218531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185569 with bad export cookie 15087238117404469004 [ 8179.467418] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8179.470517] LustreError: 218531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8179.829971] Lustre: server umount lustre-MDT0001 complete [ 8187.280253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.594902] LustreError: 222729:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8187.682941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.689765] Lustre: Skipped 4 previous similar messages [ 8191.972533] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8201.069533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.691908] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8214.043948] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8221.037558] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8228.354397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.799408] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8243.862966] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8256.481448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8256.497275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.508791] Lustre: Skipped 4 previous similar messages [ 8260.359750] Lustre: server umount lustre-OST0000 complete [ 8270.278825] Lustre: server umount lustre-MDT0000 complete [ 8273.643168] LustreError: 222710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185663 with bad export cookie 15087238117404469886 [ 8273.655543] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8273.891936] Lustre: server umount lustre-MDT0001 complete [ 8385.970627] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8388.959644] Key type lgssc unregistered [ 8389.211722] LNet: 226397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8389.218647] LNetError: 226397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8389.235893] LNet: Removed LNI 192.168.202.114@tcp [ 8390.161247] Key type .llcrypt unregistered [ 8390.162799] Key type ._llcrypt unregistered [ 8418.619545] Key type ._llcrypt registered [ 8418.622501] Key type .llcrypt registered [ 8418.741086] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8436.413550] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8437.416510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8437.583885] alg: No test for adler32 (adler32-zlib) [ 8438.690624] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 8439.019782] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8440.743176] Key type lgssc registered [ 8442.028554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8476.024479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8476.073955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.249856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8477.268396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8477.319485] Lustre: lustre-MDT0000: new disk, initializing [ 8477.375985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.393756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8481.334764] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8490.893219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.070244] Lustre: 229751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8491.143797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8491.151054] Lustre: Skipped 1 previous similar message [ 8491.259935] Lustre: lustre-MDT0001: new disk, initializing [ 8491.412842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.467994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8491.490581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8496.004387] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8505.335666] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8510.367884] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.906560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.152708] Lustre: lustre-OST0000: new disk, initializing [ 8517.157216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8517.162722] Lustre: 231723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8517.268253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.591388] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8527.394813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8527.408475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8527.548442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8532.721918] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8538.487616] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8538.711705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8544.488581] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8544.816593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8547.839154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8547.868744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.890692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.292905] Lustre: server umount lustre-OST0000 complete [ 8558.051934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8558.055524] LustreError: Skipped 1 previous similar message [ 8558.064544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.077547] Lustre: Skipped 1 previous similar message [ 8558.100296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.123289] Lustre: Skipped 2 previous similar messages [ 8563.683566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8563.693301] Lustre: Skipped 2 previous similar messages [ 8568.804410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8568.815783] Lustre: Skipped 1 previous similar message [ 8572.383496] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8572.683023] Lustre: server umount lustre-MDT0000 complete [ 8576.203166] LustreError: 229743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185966 with bad export cookie 7660256398654954433 [ 8576.207466] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.211933] LustreError: 229743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8576.531975] Lustre: server umount lustre-MDT0001 complete [ 8595.697865] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8607.005626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.664676] LustreError: 234657:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8607.696317] LustreError: 234657:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 8607.731100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.826603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8612.832958] LustreError: 234658:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8617.964355] LustreError: 234657:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8622.194598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.553457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8628.351725] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8631.261225] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8639.530268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.928670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.835551] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8660.336187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.703419] Lustre: lustre-OST0001: new disk, initializing [ 8660.705916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8660.742992] Lustre: 236966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8660.941802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8667.242652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8667.250667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8667.366922] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8667.604826] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8679.045948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8689.448584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8700.134608] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:48:08 (1786186088) [ 8703.457046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.465783] Lustre: Skipped 1 previous similar message [ 8703.472490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.477611] Lustre: Skipped 3 previous similar messages [ 8708.449893] Lustre: server umount lustre-MDT0000 complete [ 8708.582297] LustreError: 234652:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.606151] LustreError: 234652:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 8711.437168] LustreError: 234638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186101 with bad export cookie 7660256398654955315 [ 8711.455865] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8711.462647] LustreError: 234638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8711.795731] Lustre: server umount lustre-MDT0001 complete [ 8725.558526] Lustre: server umount lustre-OST0000 complete [ 8738.933877] Lustre: server umount lustre-OST0001 complete [ 8747.059989] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8756.459522] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8795.136893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.481290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8795.593874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8795.732507] Lustre: lustre-MDT0000: new disk, initializing [ 8795.882492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.916436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8800.674141] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8812.325561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.437272] Lustre: 240912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8812.488774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8812.492054] Lustre: Skipped 1 previous similar message [ 8812.542415] Lustre: lustre-MDT0001: new disk, initializing [ 8812.667533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8812.685454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8817.001441] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8827.491180] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8833.369559] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8840.298332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8840.663799] Lustre: lustre-OST0000: new disk, initializing [ 8840.670021] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8840.680591] Lustre: 242877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8842.756905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8842.766163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8842.869165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8847.658735] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8858.743154] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8864.222671] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8864.447742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8868.382537] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8868.576515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8873.440407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8873.443702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.455821] LustreError: Skipped 1 previous similar message [ 8873.466537] Lustre: Skipped 2 previous similar messages [ 8873.486979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8873.500152] Lustre: Skipped 1 previous similar message [ 8876.648485] Lustre: server umount lustre-OST0000 complete [ 8883.681854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.703882] Lustre: Skipped 1 previous similar message [ 8886.811708] Lustre: server umount lustre-MDT0000 complete [ 8888.806763] LustreError: 240918:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.828894] LustreError: 240918:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 8891.206614] LustreError: 241716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186281 with bad export cookie 7660256398654957450 [ 8891.219624] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8891.559846] Lustre: server umount lustre-MDT0001 complete [ 8900.972777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.691055] LustreError: 245098:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.745031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.751709] Lustre: Skipped 2 previous similar messages [ 8905.848972] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8914.070639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.687029] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8926.020704] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.647800] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8937.783973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.895873] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8951.443706] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9098.211700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9098.217820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.226173] Lustre: Skipped 1 previous similar message [ 9098.238252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9098.245171] Lustre: Skipped 2 previous similar messages [ 9102.805617] Lustre: server umount lustre-OST0000 complete [ 9108.960346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9108.967309] LustreError: Skipped 1 previous similar message [ 9112.477762] Lustre: server umount lustre-MDT0000 complete [ 9115.620965] LustreError: 245098:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9115.629863] LustreError: 245098:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9116.112197] LustreError: 245078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186505 with bad export cookie 7660256398654958332 [ 9116.123071] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9116.129657] LustreError: 245078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9116.573660] Lustre: server umount lustre-MDT0001 complete [ 9220.413950] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 9223.289188] Key type lgssc unregistered [ 9223.634477] LNet: 248708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9223.646688] LNetError: 248708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9224.676860] LNet: Removed LNI 192.168.202.114@tcp [ 9225.541710] Key type .llcrypt unregistered [ 9225.543962] Key type ._llcrypt unregistered [ 9250.366233] Key type ._llcrypt registered [ 9250.368428] Key type .llcrypt registered [ 9250.452152] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9266.713287] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9267.865294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9267.885721] alg: No test for adler32 (adler32-zlib) [ 9268.860547] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 9269.055258] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 9270.703295] Key type lgssc registered [ 9271.669596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9313.217136] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9322.449332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9322.507741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.783872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9323.815310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9323.885922] Lustre: lustre-MDT0000: new disk, initializing [ 9323.952559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.966993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9328.835888] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9340.535460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.704479] Lustre: 252775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.799813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9340.807417] Lustre: Skipped 1 previous similar message [ 9340.924987] Lustre: lustre-MDT0001: new disk, initializing [ 9341.056610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.138589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9341.159663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9347.175720] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9352.727681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9360.402784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9360.758445] Lustre: lustre-OST0000: new disk, initializing [ 9360.762616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9360.769077] Lustre: 254410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9360.848493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9367.108730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9367.115291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9367.242226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9367.788836] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9379.725281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.059107] Lustre: lustre-OST0001: new disk, initializing [ 9380.066630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9380.079175] Lustre: 255280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9380.219968] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9387.555943] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9389.181286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9389.189060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9389.310357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9400.169374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9407.315491] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9420.260556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.264129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.277791] Lustre: Skipped 3 previous similar messages [ 9420.292855] Lustre: Skipped 3 previous similar messages [ 9424.755662] Lustre: server umount lustre-MDT0000 complete [ 9428.052365] LustreError: 253583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186817 with bad export cookie 8013463884732014289 [ 9428.060220] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.067895] LustreError: 253583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9428.337668] Lustre: server umount lustre-MDT0001 complete [ 9442.677171] Lustre: server umount lustre-OST0000 complete [ 9457.384653] Lustre: server umount lustre-OST0001 complete [ 9475.937505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9488.083766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9500.760729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9513.847715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9534.500986] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9543.655753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.692297] Lustre: lustre-MDT0000: reset Object Index mappings [ 9543.938709] LustreError: 260307:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.951655] LustreError: 260307:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 5 previous similar messages [ 9543.995729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.023590] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9549.285845] LustreError: 260308:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.379383] LustreError: 260307:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.725714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.753718] Lustre: lustre-MDT0001: reset Object Index mappings [ 9557.243172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9561.162285] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9563.775597] Lustre: 261451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9570.401799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.653659] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9582.065225] LustreError: 261804:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9586.194693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.411385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9586.413458] Lustre: Skipped 1 previous similar message [ 9588.483779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9588.494745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9592.392760] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9599.676193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9603.158380] Lustre: 263320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9608.672756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9608.681073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.699439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9609.734916] Lustre: server umount lustre-MDT0000 complete [ 9613.424242] LustreError: 261827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187003 with bad export cookie 8013463884732021079 [ 9613.430962] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.438132] LustreError: 261827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9613.806100] Lustre: server umount lustre-MDT0001 complete [ 9628.290715] Lustre: server umount lustre-OST0000 complete [ 9631.075710] Lustre: 250284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187004/real 1786187004] req@ffff96c901f7b800 x1872952461224832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9631.118714] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.134246] Lustre: Skipped 3 previous similar messages [ 9632.056648] Lustre: server umount lustre-OST0001 complete [ 9651.207722] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:03:59 (1786187039) [ 9669.001240] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9682.940209] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9723.365626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.596952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9723.628374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9723.710882] Lustre: lustre-MDT0000: new disk, initializing [ 9723.827268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.871441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9728.373104] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9737.838951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.953920] Lustre: 267574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9737.997473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9738.000813] Lustre: Skipped 1 previous similar message [ 9738.066357] Lustre: lustre-MDT0001: new disk, initializing [ 9738.170366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9738.178113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9742.916618] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9755.250220] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9762.201140] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.228179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.600599] Lustre: lustre-OST0000: new disk, initializing [ 9769.605326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9769.613334] Lustre: 269548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9770.788587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.799191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.941155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9777.128986] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9788.910183] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9794.290759] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9794.489214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9799.756397] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9799.999800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9806.825764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.826790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9806.831563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9806.831573] Lustre: Skipped 4 previous similar messages [ 9806.841224] Lustre: Skipped 1 previous similar message [ 9806.871528] LustreError: Skipped 1 previous similar message [ 9808.568395] Lustre: server umount lustre-OST0000 complete [ 9815.008577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9815.014143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.025908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9815.035641] Lustre: Skipped 1 previous similar message [ 9822.176151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.187908] Lustre: Skipped 3 previous similar messages [ 9827.295274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9827.414807] Lustre: server umount lustre-MDT0000 complete [ 9831.228992] LustreError: 269547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187221 with bad export cookie 8013463884732023179 [ 9831.232933] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9831.238282] LustreError: 269547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9831.522969] Lustre: server umount lustre-MDT0001 complete [ 9851.218980] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9861.111993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.386463] LustreError: 272481:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9861.450135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.452629] Lustre: Skipped 2 previous similar messages [ 9866.281837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9871.840340] LustreError: 272482:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.860310] LustreError: 272482:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [ 9876.003452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.165398] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9883.942353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9890.162282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9896.430868] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9904.788646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.014931] Lustre: lustre-OST0001: new disk, initializing [ 9905.026084] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9905.036451] Lustre: 274789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9906.370995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9906.389923] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9906.555409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9911.609227] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9922.099177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9925.551844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9932.286685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9932.343482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9942.500289] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.506802] Lustre: Skipped 1 previous similar message [ 9942.510601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.513348] Lustre: Skipped 2 previous similar messages [ 9947.139904] Lustre: server umount lustre-MDT0000 complete [ 9947.629734] LustreError: 272481:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.655233] LustreError: 272481:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 3 previous similar messages [ 9951.843772] LustreError: 272463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187341 with bad export cookie 8013463884732024061 [ 9951.845215] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9951.857260] LustreError: 272463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9952.312886] Lustre: server umount lustre-MDT0001 complete [ 9966.995729] Lustre: server umount lustre-OST0000 complete [ 9969.121254] Lustre: 250287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187342/real 1786187342] req@ffff96ca052b8000 x1872952461337600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9969.181667] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.202733] Lustre: Skipped 3 previous similar messages [ 9971.974739] Lustre: server umount lustre-OST0001 complete [ 9990.078334] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10001.249328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.739412] LustreError: 277845:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.825806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.831450] Lustre: Skipped 3 previous similar messages [10007.096126] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10015.683346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10020.310960] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10023.119271] Lustre: 278985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10030.391619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10038.161958] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10046.202897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10046.267685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10051.596468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10052.589566] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10060.093929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10064.122880] Lustre: 280862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10070.835950] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10070.909667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10082.113447] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:11:10 (1786187470) [10097.636371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10097.642592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10097.672740] Lustre: Skipped 3 previous similar messages [10097.676180] Lustre: Skipped 5 previous similar messages [10103.767940] Lustre: server umount lustre-MDT0000 complete [10107.881077] LustreError: 277846:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.911979] LustreError: 277846:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 8 previous similar messages [10108.124215] LustreError: 277826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187497 with bad export cookie 8013463884732030865 [10108.136149] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.136504] LustreError: 277826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10114.649548] Lustre: server umount lustre-MDT0001 complete [10132.959333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10133.090413] Lustre: server umount lustre-OST0000 complete [10136.672941] Lustre: server umount lustre-OST0001 complete [10143.617413] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10155.054100] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10200.814803] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10211.728525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.949306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10211.971757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10212.069031] Lustre: lustre-MDT0000: new disk, initializing [10212.273523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10216.880190] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10228.192579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.454098] Lustre: 285121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10228.869897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10234.198772] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10247.507212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10247.745761] Lustre: 286751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10249.258278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10253.896684] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10266.351831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.529507] Lustre: 287624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10266.637300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10266.651286] Lustre: Skipped 6 previous similar messages [10268.398262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10268.415769] Lustre: Skipped 1 previous similar message [10268.546059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10273.272448] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10283.092149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10287.227482] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10287.230144] Lustre: Skipped 1 previous similar message [10301.555939] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:14:49 (1786187689) [10304.481150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.499260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10304.532281] Lustre: Skipped 3 previous similar messages [10304.547151] Lustre: Skipped 12 previous similar messages [10306.480896] Lustre: server umount lustre-MDT0000 complete [10309.608385] LustreError: 285132:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.627376] LustreError: 285132:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 7 previous similar messages [10310.890597] LustreError: 287623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187700 with bad export cookie 8013463884732038271 [10310.898049] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.901872] LustreError: 287623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10311.655925] Lustre: server umount lustre-MDT0001 complete [10326.271297] Lustre: server umount lustre-OST0000 complete [10331.103218] Lustre: 250284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187704/real 1786187704] req@ffff96c903083480 x1872952461487616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10331.614412] Lustre: server umount lustre-OST0001 complete [10341.684440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.723988] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10360.928061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.237522] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10377.328404] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10384.170681] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10391.314668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10397.820319] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10405.132422] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10434.467685] loop1: detected capacity change from 0 to 204800000 [10434.726621] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10434.764176] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10434.790546] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10440.398587] loop1: detected capacity change from 0 to 204800000 [10446.126319] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10446.234378] LustreError: 289859:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1087: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10446.254847] LustreError: 289859:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1087 [10446.259931] LustreError: 289859:0:(mgs_handler.c:517:mgs_target_reg()) fs1087-MDT0001: Error checking target index: rc = -13 [10446.273925] LustreError: fs1087-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10446.290949] LustreError: 293617:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10446.302335] LustreError: 293617:0:(tgt_mount.c:2129:server_put_super()) no obd fs1087-MDT0001 [10446.311649] LustreError: 293617:0:(tgt_mount.c:116:server_deregister_mount()) fs1087-MDT0001 not registered [10446.348607] Lustre: server umount fs1087-MDT0001 complete [10446.350947] LustreError: 293617:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10454.646621] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.872116] Lustre: Setting parameter fs1087-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1087-MDT0001 [10454.992371] Lustre: srv-fs1087-MDT0001: No data found on store. Initialize space. [10454.998791] Lustre: Skipped 4 previous similar messages [10455.088893] Lustre: fs1087-MDT0001: new disk, initializing [10455.093059] Lustre: Skipped 3 previous similar messages [10455.239745] Lustre: srv-fs1087-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10455.253489] Lustre: 294083:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1087-MDT0001-mdtlov: can't create new llog: rc = -115 [10455.261156] LustreError: 294083:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1087-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10459.879576] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10466.638082] Lustre: server umount fs1087-MDT0001 complete [10479.674249] loop1: detected capacity change from 0 to 204800000 [10485.239758] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10486.675226] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.198751] Lustre: 295650:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1087-MDT0001-mdtlov: can't create new llog: rc = -115 [10487.209278] LustreError: 295650:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1087-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10490.341113] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10492.904620] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10498.907344] Lustre: server umount fs1087-MDT0001 complete [10509.793515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10509.806255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.822895] Lustre: Skipped 3 previous similar messages [10509.843811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.288473] Lustre: server umount lustre-MDT0000 complete [10525.032293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10525.114209] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10530.802272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10531.006785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10539.749825] loop1: detected capacity change from 0 to 204800000 [10545.528151] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.869633] Lustre: 298095:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1087-MDT0001-mdtlov: can't create new llog: rc = -115 [10545.874937] LustreError: 298095:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1087-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10550.221864] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10555.447952] Lustre: server umount fs1087-MDT0001 complete [10566.629557] LustreError: 291703:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10566.662807] LustreError: 291703:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 102 previous similar messages [10581.475115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10587.436785] Lustre: server umount lustre-OST0000 complete [10598.194575] Lustre: server umount lustre-MDT0000 complete [10603.331250] LustreError: 289852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187993 with bad export cookie 8013463884732040959 [10603.348049] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.868088] Lustre: server umount lustre-MDT0001 complete [10618.912789] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10622.240136] Key type lgssc unregistered [10622.620902] LNet: 300444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10622.625826] LNetError: 300444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10623.655239] LNet: Removed LNI 192.168.202.114@tcp [10624.727165] Key type .llcrypt unregistered [10624.729233] Key type ._llcrypt unregistered [10651.647138] Key type ._llcrypt registered [10651.654045] Key type .llcrypt registered [10651.780292] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10653.333792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10653.370910] alg: No test for adler32 (adler32-zlib) [10654.690806] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [10654.958765] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10656.775230] Key type lgssc registered [10658.056720] Lustre: Echo OBD driver; http://www.lustre.org/ [10667.297177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10667.320181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.668231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10673.292716] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10678.259654] LustreError: 301720:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.285524] LustreError: 301720:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [10682.019221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.488294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.794444] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10695.398518] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10703.085200] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10711.202128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.903771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.960087] LustreError: 303553:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10719.215347] LustreError: 303551:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10720.271306] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10724.329287] LustreError: 303654:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.149825] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10732.014350] LustreError: 303551:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.051267] LustreError: 303551:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [10742.396139] LustreError: 303551:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.427617] LustreError: 303551:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 5 previous similar messages [10751.519491] loop1: detected capacity change from 0 to 204800000 [10751.670017] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10751.708481] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10751.752813] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10757.877514] loop1: detected capacity change from 0 to 204800000 [10762.878404] LustreError: 303654:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.891395] LustreError: 303654:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 11 previous similar messages [10764.327800] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.474806] Lustre: fs26886-OST0000: new disk, initializing [10764.479595] Lustre: srv-fs26886-OST0000: No data found on store. Initialize space. [10764.497086] Lustre: 305460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10764.628928] Lustre: fs26886-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10771.283624] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10780.772605] Lustre: server umount fs26886-OST0000 complete [10793.966609] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10796.280306] loop1: detected capacity change from 0 to 204800000 [10798.717232] LustreError: 303552:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.750453] LustreError: 303552:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 20 previous similar messages [10803.051576] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10803.535395] Lustre: fs26886-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10809.336433] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10810.301055] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10820.480620] loop2: detected capacity change from 0 to 204800000 [10820.780798] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10820.808758] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10820.821924] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10826.850297] loop2: detected capacity change from 0 to 204800000 [10832.871224] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.002366] LustreError: fs26886-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10833.014173] LustreError: fs26886-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10833.021796] LustreError: 308502:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10833.028836] LustreError: 308502:0:(tgt_mount.c:2129:server_put_super()) no obd fs26886-OST0001 [10833.033510] LustreError: 308502:0:(tgt_mount.c:116:server_deregister_mount()) fs26886-OST0001 not registered [10833.061977] Lustre: server umount fs26886-OST0001 complete [10833.070496] LustreError: 308502:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10845.370530] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10845.577960] Lustre: fs26886-OST0001: new disk, initializing [10845.581722] Lustre: srv-fs26886-OST0001: No data found on store. Initialize space. [10845.592119] Lustre: 309053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10845.669862] Lustre: fs26886-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10853.002832] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10863.632598] Lustre: server umount fs26886-OST0000 complete [10865.287108] LustreError: 307681:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.302213] LustreError: 307681:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 38 previous similar messages [10873.839444] Lustre: server umount fs26886-OST0001 complete [10887.906035] loop1: detected capacity change from 0 to 204800000 [10894.323140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.637591] Lustre: fs26886-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10903.233237] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10911.749383] loop2: detected capacity change from 0 to 204800000 [10917.948511] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.308735] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10936.864216] Lustre: server umount fs26886-OST0000 complete [10946.865566] Lustre: server umount fs26886-OST0001 complete [10947.569435] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10962.933829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10962.934256] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10962.979751] Lustre: Skipped 1 previous similar message [10962.989238] LustreError: Skipped 1 previous similar message [10962.992730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10963.025989] Lustre: Skipped 1 previous similar message [10968.033215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10968.042431] Lustre: Skipped 1 previous similar message [10973.155887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10973.160158] Lustre: Skipped 1 previous similar message [10976.223139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10976.395623] Lustre: server umount lustre-OST0000 complete [10983.394124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.401806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.404674] Lustre: Skipped 1 previous similar message [10983.419473] Lustre: Skipped 1 previous similar message [10986.340344] Lustre: server umount lustre-MDT0000 complete [10990.215596] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188380 with bad export cookie 17914082298219068159 [10990.225051] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10990.229199] LustreError: 301701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10990.552729] Lustre: server umount lustre-MDT0001 complete [11003.200584] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11006.688901] Key type lgssc unregistered [11007.010852] LNet: 314984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11007.015825] LNetError: 314984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11007.040066] LNet: Removed LNI 192.168.202.114@tcp [11008.209145] Key type .llcrypt unregistered [11008.210880] Key type ._llcrypt unregistered [11046.069425] Key type ._llcrypt registered [11046.076480] Key type .llcrypt registered [11046.182690] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [11064.058976] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11065.826486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11066.037732] alg: No test for adler32 (adler32-zlib) [11067.163960] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11067.512493] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11069.239209] Key type lgssc registered [11070.976903] Lustre: Echo OBD driver; http://www.lustre.org/ [11110.483037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11110.533620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.781554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11111.808620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11111.879948] Lustre: lustre-MDT0000: new disk, initializing [11111.943816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11111.955460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11116.611365] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11130.041357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.389947] Lustre: 318847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11130.538652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11130.548561] Lustre: Skipped 1 previous similar message [11130.680906] Lustre: lustre-MDT0001: new disk, initializing [11130.836272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11130.896037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11130.901915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11136.860789] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11149.015873] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11155.835687] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11163.217449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.612785] Lustre: lustre-OST0000: new disk, initializing [11163.627135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11163.644222] Lustre: 320815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11163.820751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11169.398501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11169.412329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11169.547414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11170.950952] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11183.713775] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11186.620453] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc [11187.795698] Lustre: Unmounted lustre-client [11200.481983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11200.484953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.497751] LustreError: Skipped 1 previous similar message [11200.510744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11200.525587] Lustre: Skipped 1 previous similar message [11202.042524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11202.060114] Lustre: Skipped 1 previous similar message [11203.886304] Lustre: server umount lustre-OST0000 complete [11212.258392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11212.266578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11212.276277] Lustre: Skipped 2 previous similar messages [11212.295848] Lustre: Skipped 1 previous similar message [11214.410129] Lustre: server umount lustre-MDT0000 complete [11218.634864] LustreError: 318837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188608 with bad export cookie 12300156003303345628 [11218.642582] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11218.647747] LustreError: 318837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11219.098962] Lustre: server umount lustre-MDT0001 complete [11232.069922] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11235.538413] Key type lgssc unregistered [11236.512285] LNet: 1 local NIs in recovery (showing 1): 192.168.202.114@tcp [11236.960485] LNet: 322978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11236.973182] LNetError: 322978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11318.891784] LNet: Removed LNI 192.168.202.114@tcp [11319.779221] Key type .llcrypt unregistered [11319.781408] Key type ._llcrypt unregistered [11341.589481] Key type ._llcrypt registered [11341.591588] Key type .llcrypt registered [11341.777910] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [11361.717291] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11364.360357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11364.437670] alg: No test for adler32 (adler32-zlib) [11365.830365] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11366.318709] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11368.071250] Key type lgssc registered [11370.015321] Lustre: Echo OBD driver; http://www.lustre.org/ [11409.523605] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:33:17 (1786188797) [11417.268562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11417.307984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.570608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11418.616749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11418.739792] Lustre: lustre-MDT0000: new disk, initializing [11418.896455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.989303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11425.384873] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11438.515103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.808713] Lustre: 326207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11438.961902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11438.966664] Lustre: Skipped 1 previous similar message [11439.140714] Lustre: lustre-MDT0001: new disk, initializing [11439.355306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11439.423963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11439.435794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11444.589547] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11455.665974] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11461.920358] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.564228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.102826] Lustre: lustre-OST0000: new disk, initializing [11471.108678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11471.120260] Lustre: 328177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11471.297270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.398314] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11479.178820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11479.192673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11479.363720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11489.744953] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11510.245906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11510.259947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11510.284930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11510.289175] Lustre: Skipped 1 previous similar message [11510.316728] Lustre: Skipped 1 previous similar message [11514.315965] Lustre: server umount lustre-OST0000 complete [11520.486737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.502846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11520.517342] Lustre: Skipped 1 previous similar message [11520.544766] Lustre: Skipped 1 previous similar message [11525.545455] Lustre: server umount lustre-MDT0000 complete [11530.724835] LustreError: 327233:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.746767] LustreError: 327233:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [11531.068798] LustreError: 327002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188920 with bad export cookie 18124820332626687856 [11531.071903] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11531.083711] LustreError: 327002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11531.540945] Lustre: server umount lustre-MDT0001 complete [11544.466379] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11548.004520] Key type lgssc unregistered [11548.303943] LNet: 330090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11548.315154] LNetError: 330090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11549.350422] LNet: Removed LNI 192.168.202.114@tcp [11550.590241] Key type .llcrypt unregistered [11550.593920] Key type ._llcrypt unregistered [11577.807352] Key type ._llcrypt registered [11577.810054] Key type .llcrypt registered [11577.977796] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11579.353809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11579.607070] alg: No test for adler32 (adler32-zlib) [11580.888924] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11581.394821] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11583.127175] Key type lgssc registered [11584.652902] Lustre: Echo OBD driver; http://www.lustre.org/ [11593.907077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11593.929308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.548275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11600.354601] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11605.989231] LustreError: 331367:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11606.033634] LustreError: 331367:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [11609.192124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.576647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11614.021499] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11621.481518] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11629.191499] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11637.291455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.693423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11645.048278] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11654.160443] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11665.472823] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11667.169832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11672.224479] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [11675.631827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.648799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11675.668060] Lustre: Skipped 2 previous similar messages [11675.675928] Lustre: Skipped 1 previous similar message [11677.304219] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [11677.321845] Lustre: Skipped 1 previous similar message [11678.420850] Lustre: server umount lustre-MDT0000 complete [11680.741659] LustreError: 331366:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.763042] LustreError: 331366:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 2 previous similar messages [11682.425063] LustreError: 332615:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.228719] LustreError: 332100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189074 with bad export cookie 12531328683794312323 [11684.243182] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11684.245182] LustreError: 332100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11685.858210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.859161] LustreError: 331363:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.881531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11689.901083] LustreError: 331363:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.902176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11689.910439] LustreError: 331363:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 1 previous similar message [11689.926602] Lustre: Skipped 1 previous similar message [11691.367143] Lustre: server umount lustre-MDT0001 complete [11695.683573] Lustre: server umount lustre-OST0000 complete [11704.899026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.734404] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11714.432558] Lustre: 335548:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.114@tcp [11717.574888] Lustre: server umount MGS complete [11729.563146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11730.296726] LustreError: 336121:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11730.421264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11736.330562] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11747.228777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.844937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11754.373059] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11765.875368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.306225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11773.107056] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11782.694524] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11784.426407] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11792.399615] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11794.820211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11801.056253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11801.060406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.088295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11801.124043] Lustre: Skipped 1 previous similar message [11815.393896] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11815.573230] Lustre: server umount lustre-OST0000 complete [11821.537215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.542604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.565788] Lustre: Skipped 1 previous similar message [11821.591910] Lustre: Skipped 6 previous similar messages [11834.336445] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11834.687735] Lustre: server umount lustre-MDT0000 complete [11836.898168] LustreError: 336853:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.919664] LustreError: 336853:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [11839.237261] LustreError: 336103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189229 with bad export cookie 12531328683794313163 [11839.242399] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11839.249667] LustreError: 336103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11839.622682] Lustre: server umount lustre-MDT0001 complete [11851.651801] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11854.807486] Key type lgssc unregistered [11855.138715] LNet: 339565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11855.157516] LNetError: 339565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11855.180534] LNet: Removed LNI 192.168.202.114@tcp [11856.345579] Key type .llcrypt unregistered [11856.350488] Key type ._llcrypt unregistered [11877.883866] Key type ._llcrypt registered [11877.894809] Key type .llcrypt registered [11878.065978] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [11895.882588] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11897.960689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11898.104904] alg: No test for adler32 (adler32-zlib) [11899.158826] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [11899.491288] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11901.184919] Key type lgssc registered [11902.464325] Lustre: Echo OBD driver; http://www.lustre.org/ [11944.271648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11944.305521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.642650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11945.680749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11945.778583] Lustre: lustre-MDT0000: new disk, initializing [11945.861046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11945.875636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11950.868318] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11963.838700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11963.994874] Lustre: 342727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11964.073976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11964.078106] Lustre: Skipped 1 previous similar message [11964.155415] Lustre: lustre-MDT0001: new disk, initializing [11964.283455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11964.318132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11964.335092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11969.193564] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11979.447846] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11986.654771] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11993.853660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.202883] Lustre: lustre-OST0000: new disk, initializing [11994.206332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11994.213314] Lustre: 344698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11994.299431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12000.887844] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12003.961678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12003.977070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12004.160071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12011.681118] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12017.666442] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12017.894742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12023.091275] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12023.513114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12029.923845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12029.931196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.934514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12029.943367] LustreError: Skipped 1 previous similar message [12029.984660] Lustre: Skipped 1 previous similar message [12032.233176] Lustre: server umount lustre-OST0000 complete [12040.164948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12040.176567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12040.181686] Lustre: Skipped 1 previous similar message [12040.208857] Lustre: Skipped 2 previous similar messages [12043.914162] Lustre: server umount lustre-MDT0000 complete [12047.786476] LustreError: 343525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189437 with bad export cookie 16836087795420053126 [12047.795000] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12047.800409] LustreError: 343525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12048.253648] Lustre: server umount lustre-MDT0001 complete [12057.626636] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:44:05 (1786189445) [12074.588429] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [12088.429813] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [12140.687768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12141.040593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12141.078164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12141.160967] Lustre: lustre-MDT0000: new disk, initializing [12141.255035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12141.271310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12146.402629] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12157.410113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.519387] Lustre: 349461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12157.571834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12157.575264] Lustre: Skipped 1 previous similar message [12157.632726] Lustre: lustre-MDT0001: new disk, initializing [12157.755239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12157.805587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12157.817918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12163.302624] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12173.270565] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12179.871885] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12187.790571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12188.149335] Lustre: lustre-OST0000: new disk, initializing [12188.162814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12188.175993] Lustre: 351433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12188.280596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12195.266280] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12197.997738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12198.011423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12198.131250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12205.566081] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12244.456202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.457077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12244.482306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12244.487783] LustreError: Skipped 1 previous similar message [12244.503028] Lustre: Skipped 1 previous similar message [12248.999269] Lustre: server umount lustre-OST0000 complete [12254.694431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.700571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12254.713296] Lustre: Skipped 2 previous similar messages [12254.724572] Lustre: Skipped 1 previous similar message [12256.986902] Lustre: server umount lustre-MDT0000 complete [12259.812670] LustreError: 349473:0:(ldlm_lib.c:1189:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.846551] LustreError: 349473:0:(ldlm_lib.c:1189:target_handle_connect()) Skipped 4 previous similar messages [12261.051358] LustreError: 349454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189650 with bad export cookie 16836087795420054008 [12261.053033] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12261.066883] LustreError: 349454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12261.504524] Lustre: server umount lustre-MDT0001 complete [12275.760555] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12279.468103] Key type lgssc unregistered [12280.123843] LNet: 353610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12280.144294] LNetError: 353610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12280.171493] LNet: Removed LNI 192.168.202.114@tcp [12281.355439] Key type .llcrypt unregistered [12281.356956] Key type ._llcrypt unregistered [12388.551374] Key type ._llcrypt registered [12388.555104] Key type .llcrypt registered [12388.695753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12389.102415] Key type .llcrypt unregistered [12389.104167] Key type ._llcrypt unregistered [12490.523439] Key type ._llcrypt registered [12490.526299] Key type .llcrypt registered [12490.701672] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12491.250438] Key type .llcrypt unregistered [12491.254078] Key type ._llcrypt unregistered [12535.724492] Key type ._llcrypt registered [12535.726668] Key type .llcrypt registered [12535.837934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12536.071669] Key type .llcrypt unregistered [12536.073537] Key type ._llcrypt unregistered [12574.811541] Key type ._llcrypt registered [12574.815834] Key type .llcrypt registered [12574.964886] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12575.325605] Key type .llcrypt unregistered [12575.328201] Key type ._llcrypt unregistered [12665.223559] Key type ._llcrypt registered [12665.225953] Key type .llcrypt registered [12665.347327] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12665.636559] Key type .llcrypt unregistered [12665.639275] Key type ._llcrypt unregistered [12697.831421] Key type ._llcrypt registered [12697.834370] Key type .llcrypt registered [12698.025228] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12698.408438] Key type .llcrypt unregistered [12698.410400] Key type ._llcrypt unregistered [12728.154290] Key type ._llcrypt registered [12728.156370] Key type .llcrypt registered [12728.260663] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12728.549448] Key type .llcrypt unregistered [12728.552914] Key type ._llcrypt unregistered [12762.905929] Key type ._llcrypt registered [12762.908970] Key type .llcrypt registered [12763.050729] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12763.396890] Key type .llcrypt unregistered [12763.399903] Key type ._llcrypt unregistered [12794.365842] Key type ._llcrypt registered [12794.368176] Key type .llcrypt registered [12794.468523] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12794.844983] Key type .llcrypt unregistered [12794.847578] Key type ._llcrypt unregistered [12836.088694] Key type ._llcrypt registered [12836.090959] Key type .llcrypt registered [12836.290947] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12836.634453] Key type .llcrypt unregistered [12836.638092] Key type ._llcrypt unregistered [12912.460270] Key type ._llcrypt registered [12912.465188] Key type .llcrypt registered [12912.558819] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12912.907420] Key type .llcrypt unregistered [12912.909675] Key type ._llcrypt unregistered [12947.826459] Key type ._llcrypt registered [12947.829211] Key type .llcrypt registered [12947.920044] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12948.189529] Key type .llcrypt unregistered [12948.193855] Key type ._llcrypt unregistered [12980.718305] Key type ._llcrypt registered [12980.727991] Key type .llcrypt registered [12980.886742] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12981.327439] Key type .llcrypt unregistered [12981.329838] Key type ._llcrypt unregistered [13042.302714] Key type ._llcrypt registered [13042.308305] Key type .llcrypt registered [13042.435125] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [13062.900995] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13065.328860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13065.641187] alg: No test for adler32 (adler32-zlib) [13067.259747] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [13067.601777] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [13069.279332] Key type lgssc registered [13071.409623] Lustre: Echo OBD driver; http://www.lustre.org/ [13114.768291] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13114.817769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13116.377958] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13116.445302] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13116.546372] Lustre: tst250fs-MDT0000: new disk, initializing [13116.720491] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13116.754179] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13122.843787] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13134.818912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13135.055918] Lustre: 374035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13135.200918] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13135.212415] Lustre: Skipped 1 previous similar message [13135.342549] Lustre: tst250fs-MDT0001: new disk, initializing [13135.553492] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13135.620522] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13135.634907] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13140.806621] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13146.767863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13155.375891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13155.865195] Lustre: tst250fs-OST0000: new disk, initializing [13155.869457] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13155.875455] Lustre: 375669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13156.087200] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13161.078606] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13161.101929] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13161.286904] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13164.537932] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13176.735276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13176.975084] Lustre: tst250fs-OST0001: new disk, initializing [13176.978784] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13176.983055] Lustre: 376542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13177.100509] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13184.926599] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13185.757202] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13185.772303] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13185.920367] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13192.822914] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13195.058553] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13195.068695] Lustre: Skipped 1 previous similar message [13216.738683] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13216.740521] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13216.751579] Lustre: Skipped 1 previous similar message [13221.954522] Lustre: server umount tst250fs-MDT0000 complete [13225.763427] LustreError: 374027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190615 with bad export cookie 17625608005683755469 [13225.765318] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13225.778585] LustreError: 374027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13226.381829] Lustre: server umount tst250fs-MDT0001 complete [13241.022333] Lustre: server umount tst250fs-OST0000 complete [13242.847287] Lustre: 372245:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190616/real 1786190616] req@ffff96ca34ef7100 x1872956444713344/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786190632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13242.860408] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13242.866328] Lustre: Skipped 2 previous similar messages [13244.944313] Lustre: server umount tst250fs-OST0001 complete [13269.409620] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [13281.165991] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13320.737764] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:05:08 (1786190708) [13323.914825] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13342.106288] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [13354.779597] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13396.050712] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13107 sec ========= 08:06:23 (1786190783) [13398.393174] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:06:26 (1786190786) === [13402.816969] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:06:30 (1786190790) === [13448.562407] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13451.486936] Key type lgssc unregistered [13451.774603] LNet: 385689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13451.782555] LNetError: 385689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13452.842074] LNet: Removed LNI 192.168.202.114@tcp [13454.302786] Key type .llcrypt unregistered [13454.314078] Key type ._llcrypt unregistered