[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502980291 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003291] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007943] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.022655] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028099] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032122] *** VALIDATE tmpfs *** [ 0.033363] *** VALIDATE proc *** [ 0.034261] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038009] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044052] debug: unmapping init [mem 0xffffffffb2e59000-0xffffffffb2e60fff] [ 0.047190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048749] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056329] rcu: Hierarchical SRCU implementation. [ 0.058502] smp: Bringing up secondary CPUs ... [ 0.059640] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.063564] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160487] node 0 deferred pages initialised in 91ms [ 0.164132] devtmpfs: initialized [ 0.165248] x86/mm: Memory block size: 128MB [ 0.168890] gcov: version magic: 0x41383552 [ 0.171297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.175084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177305] pinctrl core: initialized pinctrl subsystem [ 0.179164] [ 0.179734] ************************************************************* [ 0.182017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ** ** [ 0.187012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189012] ** ** [ 0.191016] ** This means that this kernel is built to expose internal ** [ 0.193013] ** IOMMU data structures, which may compromise security on ** [ 0.195023] ** your system. ** [ 0.197013] ** ** [ 0.200019] ** If you see this message and you are not debugging the ** [ 0.202014] ** kernel, report this immediately to your vendor! ** [ 0.204014] ** ** [ 0.206016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208013] ************************************************************* [ 0.211043] NET: Registered protocol family 16 [ 0.213515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222154] cpuidle: using governor menu [ 0.223916] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227479] PCI: Using configuration type 1 for base access [ 0.229130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.243042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.247090] cryptd: max_cpu_qlen set to 1000 [ 0.251338] ACPI: Added _OSI(Module Device) [ 0.253025] ACPI: Added _OSI(Processor Device) [ 0.254020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.256019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.261349] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.269359] ACPI: Interpreter enabled [ 0.271091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273018] ACPI: Using IOAPIC for interrupt routing [ 0.275156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279508] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.288570] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298173] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303385] acpiphp: Slot [2] registered [ 0.305104] acpiphp: Slot [5] registered [ 0.306097] acpiphp: Slot [6] registered [ 0.307238] acpiphp: Slot [7] registered [ 0.308109] acpiphp: Slot [8] registered [ 0.310140] acpiphp: Slot [9] registered [ 0.311223] acpiphp: Slot [10] registered [ 0.313165] acpiphp: Slot [3] registered [ 0.314085] acpiphp: Slot [4] registered [ 0.315109] acpiphp: Slot [11] registered [ 0.317138] acpiphp: Slot [12] registered [ 0.319210] acpiphp: Slot [13] registered [ 0.320094] acpiphp: Slot [14] registered [ 0.321076] acpiphp: Slot [15] registered [ 0.322082] acpiphp: Slot [16] registered [ 0.324119] acpiphp: Slot [17] registered [ 0.325105] acpiphp: Slot [18] registered [ 0.326063] acpiphp: Slot [19] registered [ 0.327076] acpiphp: Slot [20] registered [ 0.329092] acpiphp: Slot [21] registered [ 0.330085] acpiphp: Slot [22] registered [ 0.331064] acpiphp: Slot [23] registered [ 0.332063] acpiphp: Slot [24] registered [ 0.333163] acpiphp: Slot [25] registered [ 0.335114] acpiphp: Slot [26] registered [ 0.336167] acpiphp: Slot [27] registered [ 0.337304] acpiphp: Slot [28] registered [ 0.338087] acpiphp: Slot [29] registered [ 0.339072] acpiphp: Slot [30] registered [ 0.340062] acpiphp: Slot [31] registered [ 0.341029] PCI host bridge to bus 0000:00 [ 0.342035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.350042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358290] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361115] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364460] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373039] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378909] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.381020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387423] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.392420] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398951] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.434848] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.442026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.468034] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.479288] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.496018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.552019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.571792] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.582030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.596031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.627029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.648390] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.658019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.669019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.704021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.717570] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.734022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.757022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.767940] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.783019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.813019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.824882] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.827455] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.830437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.832397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.835248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.841032] iommu: Default domain type: Passthrough [ 0.843530] SCSI subsystem initialized [ 0.846115] ACPI: bus type USB registered [ 0.848144] usbcore: registered new interface driver usbfs [ 0.850155] usbcore: registered new interface driver hub [ 0.852104] usbcore: registered new device driver usb [ 0.853222] pps_core: LinuxPPS API ver. 1 registered [ 0.854012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.856082] PTP clock support registered [ 0.860099] EDAC MC: Ver: 3.0.0 [ 0.862162] PCI: Using ACPI for IRQ routing [ 0.864000] NetLabel: Initializing [ 0.866019] NetLabel: domain hash size = 128 [ 0.867015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.869112] NetLabel: unlabeled traffic allowed by default [ 0.872204] vgaarb: loaded [ 0.873330] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.876021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.881341] clocksource: Switched to clocksource kvm-clock [ 0.977801] VFS: Disk quotas dquot_6.6.0 [ 0.979459] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982356] *** VALIDATE ramfs *** [ 0.983721] *** VALIDATE hugetlbfs *** [ 0.985411] pnp: PnP ACPI init [ 0.987878] pnp: PnP ACPI: found 6 devices [ 1.004520] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.008026] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.009775] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.011592] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.013841] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.016694] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.019707] NET: Registered protocol family 2 [ 1.022649] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.027930] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.032120] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.037523] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.041293] TCP: Hash tables configured (established 65536 bind 65536) [ 1.044404] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.047823] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.051415] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.054805] NET: Registered protocol family 1 [ 1.057677] RPC: Registered named UNIX socket transport module. [ 1.059176] RPC: Registered udp transport module. [ 1.060319] RPC: Registered tcp transport module. [ 1.061478] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.063221] NET: Registered protocol family 44 [ 1.064302] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.065613] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.066992] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.068481] PCI: CLS 0 bytes, default 64 [ 1.070281] Unpacking initramfs... [ 2.493836] debug: unmapping init [mem 0xffffa05f3cc54000-0xffffa05f3ffbffff] [ 2.497808] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.500144] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.503247] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.007283] Initialise system trusted keyrings [ 3.009358] Key type blacklist registered [ 3.011317] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.020574] zbud: loaded [ 3.024210] *** VALIDATE nfs *** [ 3.025469] *** VALIDATE nfs4 *** [ 3.027191] pstore: using deflate compression [ 3.031235] Platform Keyring initialized [ 3.137604] NET: Registered protocol family 38 [ 3.144246] Key type asymmetric registered [ 3.145768] Asymmetric key parser 'x509' registered [ 3.147742] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.151230] io scheduler mq-deadline registered [ 3.153392] io scheduler kyber registered [ 3.155730] io scheduler bfq registered [ 3.158380] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.162692] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.166192] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.169873] ACPI: Power Button [PWRF] [ 3.175914] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.181674] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.206703] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.218784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.241017] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.269814] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.300102] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.305079] Non-volatile memory driver v1.3 [ 3.306987] Linux agpgart interface v0.103 [ 3.336631] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.339401] vda: detected capacity change from 0 to 75083776 [ 3.356795] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.359414] vdb: detected capacity change from 0 to 1073741824 [ 3.381474] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.384205] vdc: detected capacity change from 0 to 2621440000 [ 3.408207] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.411057] vdd: detected capacity change from 0 to 2621440000 [ 3.429786] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.433242] vde: detected capacity change from 0 to 4294967296 [ 3.455331] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.458473] vdf: detected capacity change from 0 to 4294967296 [ 3.469393] libphy: Fixed MDIO Bus: probed [ 3.494334] usbcore: registered new interface driver usbserial_generic [ 3.497022] usbserial: USB Serial support registered for generic [ 3.499800] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.504441] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.506523] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.509261] mousedev: PS/2 mouse device common for all mice [ 3.513931] rtc_cmos 00:05: RTC can wake from S4 [ 3.517249] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.520991] rtc_cmos 00:05: registered as rtc0 [ 3.525622] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.526772] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.532688] intel_pstate: CPU model not supported [ 3.539146] hid: raw HID events driver (C) Jiri Kosina [ 3.540788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.542049] usbcore: registered new interface driver usbhid [ 3.547051] usbhid: USB HID core driver [ 3.548628] drop_monitor: Initializing network drop monitor service [ 3.551353] Initializing XFRM netlink socket [ 3.553492] NET: Registered protocol family 10 [ 3.556375] Segment Routing with IPv6 [ 3.557932] NET: Registered protocol family 17 [ 3.559874] mpls_gso: MPLS GSO support [ 3.566305] RAS: Correctable Errors collector initialized. [ 3.568529] AVX version of gcm_enc/dec engaged. [ 3.570293] AES CTR mode by8 optimization enabled [ 3.647931] sched_clock: Marking stable (3647843206, 0)->(4555255910, -907412704) [ 3.652689] registered taskstats version 1 [ 3.655664] Loading compiled-in X.509 certificates [ 3.657715] zswap: loaded using pool lzo/zbud [ 3.679950] Key type big_key registered [ 3.692868] Key type encrypted registered [ 3.694469] ima: No TPM chip found, activating TPM-bypass! [ 3.696538] ima: Allocated hash algorithm: sha1 [ 3.698408] ima: No architecture policies found [ 3.700242] evm: Initialising EVM extended attributes: [ 3.702241] evm: security.selinux [ 3.703473] evm: security.ima [ 3.704607] evm: security.capability [ 3.706009] evm: HMAC attrs: 0x1 [ 3.708552] rtc_cmos 00:05: setting system clock to 2026-09-03 19:24:53 UTC (1788463493) [ 3.715194] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.718045] debug: unmapping init [mem 0xffffffffb2b82000-0xffffffffb2e58fff] [ 3.728477] Write protecting the kernel read-only data: 28672k [ 3.731618] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 3.734351] debug: unmapping init [mem 0xffffffffb1b14000-0xffffffffb1bfffff] [ 3.768918] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.776367] systemd[1]: Detected virtualization kvm. [ 3.778291] systemd[1]: Detected architecture x86-64. [ 3.780182] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.806462] systemd[1]: No hostname configured. [ 3.807940] systemd[1]: Set hostname to . [ 3.809680] random: systemd: uninitialized urandom read (16 bytes read) [ 3.811808] systemd[1]: Initializing machine ID from random generator. [ 3.837247] random: ln: uninitialized urandom read (6 bytes read) [ 3.943637] random: systemd: uninitialized urandom read (16 bytes read) [ 3.946030] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.951522] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.954826] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.538492] device-mapper: uevent: version 1.0.3 [ 4.540298] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.933930] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.271193] virtio_net virtio0 ens2: renamed from eth0 [ 5.284600] scsi host0: ata_piix [ 5.331954] scsi host1: ata_piix [ 5.333930] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.337092] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.609436] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.234395] random: crng init done [ 10.235832] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.576270] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.698945] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.961390] SELinux: Disabled at runtime. [ 12.082974] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.091514] systemd[1]: Detected virtualization kvm. [ 12.093483] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.581386] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.584436] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.589792] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.593935] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.598235] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.605592] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.610513] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ 12.664353] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.103988] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.380120] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.419341] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.532550] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.544987] EDAC sbridge: Ver: 1.1.2 [ 14.885945] Key type dns_resolver registered [ 15.210913] NFS: Registering the id_resolver key type [ 15.212906] Key type id_resolver registered [ 15.214473] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg110-server login: [ 49.939286] hrtimer: interrupt took 8305824 ns [ 74.577893] libcfs: loading out-of-tree module taints kernel. [ 74.623989] Key type ._llcrypt registered [ 74.625400] Key type .llcrypt registered [ 74.725861] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 95.676720] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 97.801144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.832906] alg: No test for adler32 (adler32-zlib) [ 99.302680] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 100.355921] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 102.039184] Key type lgssc registered [ 104.604906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.930540] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.512616] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 187.024293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.103526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.457881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.505614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.640850] Lustre: lustre-MDT0000: new disk, initializing [ 188.829091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.844502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 195.124203] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.286959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.437417] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.492368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.496142] Lustre: Skipped 1 previous similar message [ 211.596819] Lustre: lustre-MDT0001: new disk, initializing [ 211.650392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.670708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.674453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.888985] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.382647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.778847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.025365] Lustre: lustre-OST0000: new disk, initializing [ 233.028616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.036246] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.108804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.766180] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.709676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.718833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.811621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.128809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.230926] Lustre: lustre-OST0001: new disk, initializing [ 254.233963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.238961] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.330970] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.914413] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.233394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.240959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.359911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.138477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.797954] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.665398] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.535390] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing yml_node [ 300.734384] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 303.544959] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 306.131394] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 307.876401] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 15:29:55 EDT 2026 [ 326.124188] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 327.792862] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.056319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.066818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.090198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.597494] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.597776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.635792] Lustre: Skipped 1 previous similar message [ 346.646354] Lustre: Skipped 3 previous similar messages [ 351.712423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.734263] Lustre: Skipped 3 previous similar messages [ 356.849711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.857576] Lustre: Skipped 3 previous similar messages [ 358.367211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.962302] Lustre: server umount lustre-MDT0000 complete [ 363.759983] LustreError: 8451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463853 with bad export cookie 15303088665299888709 [ 363.763435] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.772427] LustreError: 8451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 367.072372] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.072854] LustreError: 12155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.076820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.098545] Lustre: Skipped 2 previous similar messages [ 367.116735] LustreError: 12155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 368.096388] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.109382] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 373.223163] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.270671] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 378.338474] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.343094] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.343101] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.343110] Lustre: Skipped 5 previous similar messages [ 378.386440] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 378.678830] Lustre: server umount lustre-MDT0001 complete [ 383.739937] Lustre: server umount lustre-OST0000 complete [ 388.457703] Lustre: server umount lustre-OST0001 complete [ 396.570895] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 407.104606] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 445.272057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.481843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.506884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.574611] Lustre: lustre-MDT0000: new disk, initializing [ 445.639450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.652380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.922161] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 460.169904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.530205] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.775210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.780199] Lustre: Skipped 1 previous similar message [ 460.903711] Lustre: lustre-MDT0001: new disk, initializing [ 461.021146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.083169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.102708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.719575] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 483.449424] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.558883] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 498.828535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.115373] Lustre: lustre-OST0000: new disk, initializing [ 499.118039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.124071] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.181798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.850897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.869541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.010133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.468746] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 516.639234] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.572387] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.761945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 527.028252] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 527.223767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.937505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.940627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.944641] LustreError: Skipped 1 previous similar message [ 531.954647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.965412] Lustre: Skipped 1 previous similar message [ 531.980450] Lustre: Skipped 1 previous similar message [ 535.670657] Lustre: server umount lustre-OST0000 complete [ 542.179776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.212030] Lustre: server umount lustre-MDT0000 complete [ 547.306921] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.339681] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 550.583333] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464040 with bad export cookie 15303088665299890242 [ 550.586506] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.596851] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.982643] Lustre: server umount lustre-MDT0001 complete [ 557.272962] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:34:04 (1788464044) [ 578.858360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.409751] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.531269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.333512] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 594.895864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.259231] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 608.884305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.776641] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.597527] Lustre: Failing over lustre-MDT0000 [ 618.936068] Lustre: server umount lustre-MDT0000 complete [ 619.620440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.643269] Lustre: Skipped 1 previous similar message [ 619.649479] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.678826] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 635.295870] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464109/real 1788464109] req@ffffa05fbf39d500 x1875339997512704/t0(0) o400->MGC192.168.201.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788464125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 635.345773] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.817498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.600860] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa05e867e3480 x1875339997515264/t0(0) o250->MGC192.168.201.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 645.983681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.991330] Lustre: Skipped 1 previous similar message [ 646.082129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 651.239265] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 651.249521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 651.287231] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 651.361354] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 662.990379] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 669.666935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.691533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 669.693139] Lustre: Skipped 2 previous similar messages [ 669.708779] Lustre: Skipped 4 previous similar messages [ 674.270583] Lustre: server umount lustre-MDT0000 complete [ 674.784979] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.803396] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 679.042543] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464168 with bad export cookie 15303088665299891656 [ 679.054500] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.059694] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 679.411495] Lustre: server umount lustre-MDT0001 complete [ 688.743272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.518348] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 704.936371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.787089] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 722.904780] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.012921] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 732.915411] Lustre: Failing over lustre-MDT0000 [ 733.502518] Lustre: server umount lustre-MDT0000 complete [ 736.227707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.240814] Lustre: Skipped 1 previous similar message [ 736.244322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 741.344601] LustreError: 23266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.353225] LustreError: 23266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 752.351324] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788464226/real 1788464226] req@ffffa05fbf39e680 x1875339997544448/t0(0) o400->MGC192.168.201.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788464242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.400106] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.133295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.854542] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa05fbf39ca80 x1875339997547392/t0(0) o250->MGC192.168.201.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 763.499512] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 768.491632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 768.492789] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 768.506505] Lustre: Skipped 1 previous similar message [ 768.541594] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 769.577858] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 782.851907] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 788.960743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 788.990993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.271232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 802.694385] Lustre: server umount lustre-MDT0000 complete [ 807.635741] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464297 with bad export cookie 15303088665299892594 [ 807.638861] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.646613] LustreError: 23247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 807.957105] Lustre: server umount lustre-MDT0001 complete [ 818.316631] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:38:25 (1788464305) [ 837.289841] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 846.407175] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 881.769557] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 894.481608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.816268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 894.856609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 894.948001] Lustre: lustre-MDT0000: new disk, initializing [ 895.063161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.068115] Lustre: Skipped 3 previous similar messages [ 895.080367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 900.054510] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 913.066856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.180443] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 913.423543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 918.368824] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 929.277803] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 935.753569] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 943.211939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 943.617245] Lustre: 31615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 945.472735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 945.511441] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 945.671706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 945.701637] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 950.665795] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 960.881993] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.113778] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 991.715681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.721571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.740481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.745439] Lustre: Skipped 3 previous similar messages [ 991.774169] Lustre: Skipped 7 previous similar messages [ 996.613625] Lustre: server umount lustre-OST0000 complete [ 1000.416403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1000.422226] LustreError: Skipped 1 previous similar message [ 1006.700405] Lustre: server umount lustre-MDT0000 complete [ 1007.078019] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.094333] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 1010.661363] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464500 with bad export cookie 15303088665299893000 [ 1010.667226] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.667538] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1011.369179] Lustre: server umount lustre-MDT0001 complete [ 1022.533144] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1025.615063] Key type lgssc unregistered [ 1025.861587] LNet: 33593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.877508] LNetError: 33593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.919198] LNet: Removed LNI 192.168.201.110@tcp [ 1028.245132] Key type .llcrypt unregistered [ 1028.247091] Key type ._llcrypt unregistered [ 1054.312552] Key type ._llcrypt registered [ 1054.314901] Key type .llcrypt registered [ 1054.387567] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1055.787694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.858799] alg: No test for adler32 (adler32-zlib) [ 1056.997502] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1057.335051] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1059.047179] Key type lgssc registered [ 1060.197973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.787636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.811793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.259236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.185990] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1080.805210] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.819801] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1085.576916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.868885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.593503] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1099.121884] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.744388] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1115.273116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.574951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.094803] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1130.942715] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1131.977751] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1147.033710] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:43:54 (1788464634) [ 1148.804313] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1151.065851] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1159.532721] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:44:06 (1788464646) [ 1168.716918] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:44:15 (1788464655) [ 1169.701914] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1182.862361] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1182.868639] Lustre: Skipped 2 previous similar messages [ 1186.696066] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:44:32 (1788464672) [ 1190.719249] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1190.742988] Lustre: Skipped 2 previous similar messages [ 1201.928573] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1201.934906] Lustre: Skipped 1 previous similar message [ 1225.282592] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1225.290511] Lustre: Skipped 5 previous similar messages [ 1231.203099] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:45:17 (1788464717) [ 1234.875573] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1238.854676] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1250.511932] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:45:37 (1788464737) [ 1258.081650] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 1258.111198] Lustre: Skipped 6 previous similar messages [ 1270.109368] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:45:56 (1788464756) [ 1289.698368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.700801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.705918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.714625] Lustre: Skipped 1 previous similar message [ 1289.725240] Lustre: Skipped 1 previous similar message [ 1293.025267] Lustre: server umount lustre-OST0000 complete [ 1299.942725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.944917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.953050] Lustre: Skipped 1 previous similar message [ 1302.690470] Lustre: server umount lustre-MDT0000 complete [ 1305.074842] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.117666] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1307.765117] LustreError: 38564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464797 with bad export cookie 12115671616838246125 [ 1307.766409] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.783695] LustreError: 38564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1308.326695] Lustre: server umount lustre-MDT0001 complete [ 1323.280905] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1327.736594] Key type lgssc unregistered [ 1328.240618] LNet: 42864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.249133] LNetError: 42864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.283220] LNet: Removed LNI 192.168.201.110@tcp [ 1329.539520] Key type .llcrypt unregistered [ 1329.545389] Key type ._llcrypt unregistered [ 1349.419902] Key type ._llcrypt registered [ 1349.425360] Key type .llcrypt registered [ 1349.571529] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1366.010862] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1367.038404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.071506] alg: No test for adler32 (adler32-zlib) [ 1368.160887] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1368.497132] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1370.127191] Key type lgssc registered [ 1371.201293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.661506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.690977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.050253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1416.104573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1416.189991] Lustre: lustre-MDT0000: new disk, initializing [ 1416.314701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.356862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1421.443493] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1433.105367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.237791] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1433.295930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1433.299215] Lustre: Skipped 1 previous similar message [ 1433.352659] Lustre: lustre-MDT0001: new disk, initializing [ 1433.450479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.494288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1433.515117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1438.710704] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1449.049077] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1456.561294] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1464.965610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.427541] Lustre: lustre-OST0000: new disk, initializing [ 1465.433313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1465.455980] Lustre: 47983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1465.622877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.079660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1471.088416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1471.247375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1472.387346] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1485.263724] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.450636] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1494.696335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1500.284144] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1500.554187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1507.296911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.306636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.316189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.332748] Lustre: Skipped 1 previous similar message [ 1507.354274] Lustre: Skipped 1 previous similar message [ 1508.615899] Lustre: server umount lustre-OST0000 complete [ 1515.492347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1515.502841] LustreError: Skipped 1 previous similar message [ 1515.504649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.515168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.543089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.545106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.632526] Lustre: server umount lustre-MDT0000 complete [ 1527.027089] LustreError: 46006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465016 with bad export cookie 18264429029680907622 [ 1527.041031] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.059471] LustreError: 46006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1527.723380] Lustre: server umount lustre-MDT0001 complete [ 1540.607547] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:50:27 (1788465027) [ 1548.289261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.787071] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.800878] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1548.891860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.898731] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.201937] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1559.014474] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.130142] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.652342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.282228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1571.605460] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1580.190926] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1589.284602] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1597.979521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1598.209660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.375855] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1615.314578] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1624.401396] LustreError: 52975:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1625.298274] Lustre: Modifying parameter general.timeout=1 in log params [ 1626.321964] Lustre: Modifying parameter general.timeout=2 in log params [ 1628.977516] Lustre: Modifying parameter general.timeout=5 in log params [ 1628.986609] Lustre: Skipped 2 previous similar messages [ 1632.998157] Lustre: Modifying parameter general.timeout=9 in log params [ 1633.001737] Lustre: Skipped 3 previous similar messages [ 1641.570528] Lustre: Modifying parameter general.timeout=18 in log params [ 1641.578339] Lustre: Skipped 8 previous similar messages [ 1658.323153] Lustre: Modifying parameter general.timeout=36 in log params [ 1658.328873] Lustre: Skipped 17 previous similar messages [ 1690.978580] Lustre: Modifying parameter general.timeout=64 in log params [ 1690.985605] Lustre: Skipped 27 previous similar messages [ 1755.883317] Lustre: Modifying parameter general.timeout=116 in log params [ 1755.887406] Lustre: Skipped 51 previous similar messages [ 1784.276463] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:54:31 (1788465271) [ 1789.343485] Lustre: 59940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1792.276627] Lustre: 60039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1795.232532] Lustre: 60149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1801.245112] Lustre: 60357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1801.255538] Lustre: 60357:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1841.441247] Lustre: 61830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1841.457681] Lustre: 61830:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1841.482886] LustreError: 61830:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1845.605792] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:55:33 (1788465333) [ 1861.091509] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.096786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.108206] Lustre: Skipped 1 previous similar message [ 1861.136264] Lustre: Skipped 3 previous similar messages [ 1866.210483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.216817] Lustre: Skipped 2 previous similar messages [ 1867.445952] Lustre: server umount lustre-MDT0000 complete [ 1871.328861] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.363504] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1872.086535] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465361 with bad export cookie 18264429029680908504 [ 1872.088667] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.101241] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1872.346533] Lustre: server umount lustre-MDT0001 complete [ 1887.536715] Lustre: server umount lustre-OST0000 complete [ 1915.460389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.807439] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.840449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1915.852904] Lustre: Skipped 36 previous similar messages [ 1916.078369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.336378] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1931.457216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.626645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1931.716767] Lustre: 64363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1936.634169] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1945.718169] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.087677] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1959.579469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.769563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1960.091944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.101893] Lustre: Skipped 1 previous similar message [ 1968.234861] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1977.149316] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.144119] Lustre: 66938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1988.578817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.589390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1988.614058] Lustre: Skipped 1 previous similar message [ 1988.617937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.658489] Lustre: Skipped 1 previous similar message [ 1993.625608] Lustre: server umount lustre-OST0000 complete [ 2003.940465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.946457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.962196] Lustre: Skipped 2 previous similar messages [ 2003.989227] LustreError: Skipped 1 previous similar message [ 2006.135417] Lustre: server umount lustre-MDT0000 complete [ 2009.065454] LustreError: 65181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.085979] LustreError: 65181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2011.344596] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465501 with bad export cookie 18264429029680911864 [ 2011.352584] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.353531] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2011.724343] Lustre: server umount lustre-MDT0001 complete [ 2024.510172] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2027.513335] Key type lgssc unregistered [ 2027.822644] LNet: 68148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.831286] LNetError: 68148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.846983] LNet: Removed LNI 192.168.201.110@tcp [ 2028.641098] Key type .llcrypt unregistered [ 2028.643467] Key type ._llcrypt unregistered [ 2051.753315] Key type ._llcrypt registered [ 2051.754937] Key type .llcrypt registered [ 2051.847517] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2053.063691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.309419] alg: No test for adler32 (adler32-zlib) [ 2054.257289] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2054.538034] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2056.191245] Key type lgssc registered [ 2057.183722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.717556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2065.751387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.399510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.879864] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2077.673311] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.687644] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2081.611559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.436701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.591581] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2098.163477] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.535563] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2114.602831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2115.269183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.947068] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2128.760072] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2147.488065] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2158.560811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.578935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.594093] Lustre: Skipped 2 previous similar messages [ 2158.603235] Lustre: Skipped 2 previous similar messages [ 2163.052856] Lustre: server umount lustre-MDT0000 complete [ 2163.685869] LustreError: 70290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.718124] LustreError: 70290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2167.547333] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465657 with bad export cookie 15289617620974941650 [ 2167.555456] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.559125] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.065296] Lustre: server umount lustre-MDT0001 complete [ 2183.799645] Lustre: server umount lustre-OST0000 complete [ 2211.020818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.371105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2211.442644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2211.934679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.568614] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2225.747982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.805218] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2225.828882] Lustre: 74463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2226.135285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.053264] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2238.363622] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2245.159529] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2254.367986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.665442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2254.987659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.623528] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2267.590360] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.224992] Lustre: 77038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2271.406329] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2280.421935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.428588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.465735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.481702] Lustre: Skipped 1 previous similar message [ 2282.294187] Lustre: server umount lustre-OST0000 complete [ 2290.657037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.685662] Lustre: Skipped 1 previous similar message [ 2290.709643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.717767] Lustre: Skipped 1 previous similar message [ 2294.420834] Lustre: server umount lustre-MDT0000 complete [ 2295.793641] LustreError: 75293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.807502] LustreError: 75293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2298.659060] LustreError: 74455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465788 with bad export cookie 15289617620974942483 [ 2298.666612] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.678940] LustreError: 74455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2299.248538] Lustre: server umount lustre-MDT0001 complete [ 2309.119415] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2311.611532] Key type lgssc unregistered [ 2311.959923] LNet: 78250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.973455] LNetError: 78250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.995667] LNet: Removed LNI 192.168.201.110@tcp [ 2312.789977] Key type .llcrypt unregistered [ 2312.793440] Key type ._llcrypt unregistered [ 2336.214686] Key type ._llcrypt registered [ 2336.217808] Key type .llcrypt registered [ 2336.428318] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2338.016511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.085071] alg: No test for adler32 (adler32-zlib) [ 2339.265601] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2339.584340] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2341.351670] Key type lgssc registered [ 2343.193996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2351.657379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2351.675054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.104753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.613826] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2362.338263] LustreError: 79661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.354438] LustreError: 79661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2366.294047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.914576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.544100] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2379.555277] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.115764] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2392.021882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.640977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.589661] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2407.071584] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2417.400611] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2418.323806] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2419.396962] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2421.976063] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2421.978161] Lustre: Skipped 2 previous similar messages [ 2426.378428] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2426.383958] Lustre: Skipped 4 previous similar messages [ 2434.608201] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2434.616953] Lustre: Skipped 8 previous similar messages [ 2451.445217] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2451.454654] Lustre: Skipped 15 previous similar messages [ 2483.547420] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2483.554197] Lustre: Skipped 35 previous similar messages [ 2529.949459] Lustre: 87763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2533.126568] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:07:00 (1788466020) [ 2535.785237] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2539.892897] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:07:05 (1788466025) [ 2546.145599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.167176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.212050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2550.240300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2550.249194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.260694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2550.295369] Lustre: Skipped 1 previous similar message [ 2552.737876] Lustre: server umount lustre-OST0000 complete [ 2560.480847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2560.487329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2560.513884] Lustre: Skipped 1 previous similar message [ 2565.602419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.605096] Lustre: Skipped 1 previous similar message [ 2566.779167] Lustre: server umount lustre-MDT0000 complete [ 2570.721779] LustreError: 81511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.753571] LustreError: 81511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2571.504181] LustreError: 79643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466061 with bad export cookie 18101439099696328034 [ 2571.506728] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.515202] LustreError: 79643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2572.593615] Lustre: server umount lustre-MDT0001 complete [ 2586.442153] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2589.989526] Key type lgssc unregistered [ 2590.375583] LNet: 89013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.379890] LNetError: 89013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.413258] LNet: Removed LNI 192.168.201.110@tcp [ 2591.970395] Key type .llcrypt unregistered [ 2591.977803] Key type ._llcrypt unregistered [ 2598.526261] Key type ._llcrypt registered [ 2598.527541] Key type .llcrypt registered [ 2598.615339] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2602.588736] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules [ 2603.229550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.249108] alg: No test for adler32 (adler32-zlib) [ 2604.501079] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2604.987648] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2606.744111] Key type lgssc registered [ 2608.813467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2610.858574] LustreError: 90212:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2629.199686] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2632.634947] LustreError: 90212:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2639.749272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2639.763463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.097633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.446880] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2650.607500] LustreError: 91210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.623328] LustreError: 91210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2655.712938] LustreError: 91211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.874778] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:09:03 (1788466143) [ 2660.834502] LustreError: 91210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.518994] LustreError: 91242:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2661.526340] LustreError: 91242:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2661.529828] LustreError: 91242:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2661.885174] Lustre: server umount lustre-MDT0000 complete [ 2684.279202] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2687.534741] Key type lgssc unregistered [ 2687.920721] LNet: 92757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.929341] LNetError: 92757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.965085] LNet: Removed LNI 192.168.201.110@tcp [ 2688.925358] Key type .llcrypt unregistered [ 2688.928493] Key type ._llcrypt unregistered [ 2708.770666] Key type ._llcrypt registered [ 2708.773501] Key type .llcrypt registered [ 2709.042680] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2710.515826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2710.675686] alg: No test for adler32 (adler32-zlib) [ 2712.001590] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2712.406300] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2714.129639] Key type lgssc registered [ 2715.575271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2726.434213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2726.474287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.428495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.748732] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2738.682571] LustreError: 93976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.709923] LustreError: 93976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2743.778687] LustreError: 93975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.189064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.957105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2750.309114] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2758.155443] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2764.393448] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2772.015508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.812307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.332522] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2786.708096] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2835.350691] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:12:03 (1788466323) [ 2836.917732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2838.867656] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:12:06 (1788466326) [ 2845.668715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.671413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.686301] Lustre: Skipped 2 previous similar messages [ 2845.715140] Lustre: Skipped 2 previous similar messages [ 2848.886742] Lustre: server umount lustre-MDT0000 complete [ 2850.789038] LustreError: 94968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2850.819256] LustreError: 94968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2852.504970] LustreError: 93957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466342 with bad export cookie 5100385787477531281 [ 2852.516504] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.528586] LustreError: 93957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2852.848154] Lustre: server umount lustre-MDT0001 complete [ 2867.541509] Lustre: server umount lustre-OST0000 complete [ 2880.117935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.733732] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.055566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.923545] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2886.123303] LustreError: 98274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.154618] LustreError: 98274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2894.675282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.407734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2900.345545] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2908.746485] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2918.149070] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2933.748624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.954934] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2933.962355] LustreError: 98262:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2933.982537] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2933.988585] LustreError: 100367:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2933.993972] LustreError: 100367:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2933.999557] LustreError: 100367:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2934.149035] Lustre: server umount lustre-OST0000 complete [ 2934.152292] LustreError: 100367:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2942.999690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.224290] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2943.236231] LustreError: 98262:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2943.264639] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2943.269164] LustreError: 100828:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2943.272741] LustreError: 100828:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2943.275921] LustreError: 100828:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2943.314388] Lustre: server umount lustre-OST0000 complete [ 2943.316783] LustreError: 100828:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2953.307467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.544615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2953.547358] Lustre: Found index 0 for lustre-OST0000, updating log [ 2953.552188] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2953.611757] Lustre: lustre-OST0000: new disk, initializing [ 2953.615338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2953.625330] Lustre: 101368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2953.884854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.928572] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2964.467956] Lustre: Failing over lustre-OST0000 [ 2964.617308] Lustre: server umount lustre-OST0000 complete [ 2970.592836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.595664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.603740] Lustre: Skipped 1 previous similar message [ 2970.618427] Lustre: Skipped 1 previous similar message [ 2975.714936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.725288] Lustre: Skipped 1 previous similar message [ 2980.833105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.843737] Lustre: Skipped 1 previous similar message [ 2982.368518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2982.704209] Lustre: server umount lustre-MDT0000 complete [ 2985.956458] LustreError: 99013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.985720] LustreError: 99013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2986.219165] LustreError: 98253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466476 with bad export cookie 5100385787477534025 [ 2986.222917] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.228317] LustreError: 98253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2986.617781] Lustre: server umount lustre-MDT0001 complete [ 2996.537786] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:14:43 (1788466483) [ 3018.646737] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3029.547402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.037462] LustreError: 103721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.173275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.625739] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3046.263112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.515674] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3056.355357] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3064.267158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.186991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.192415] Lustre: Skipped 1 previous similar message [ 3073.527642] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3083.283456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.408297] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3083.546668] Lustre: lustre-OST0001: new disk, initializing [ 3083.550635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3083.555989] Lustre: 106037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3091.160900] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3093.660586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3093.681590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3093.819255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3099.792935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3106.127798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3113.441250] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.445653] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.471288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3117.047678] Lustre: server umount lustre-MDT0001 complete [ 3119.597052] LustreError: 103722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.618802] LustreError: 103722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3127.635135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.787391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3127.789724] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3127.837637] Lustre: 103711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3129.726136] LustreError: 107618:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3129.733534] LustreError: 107618:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3129.743241] LustreError: 103714:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3129.855224] LustreError: 107624:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 3129.878132] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3129.878132] [ 3133.317330] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3133.425606] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3145.305065] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:17:12 (1788466632) [ 3165.714069] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3185.194478] Lustre: 109654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3209.046720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3212.597804] Lustre: 110791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3297.248495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3297.249947] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.250625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3297.250630] Lustre: Skipped 4 previous similar messages [ 3297.288246] Lustre: Skipped 5 previous similar messages [ 3301.161941] Lustre: server umount lustre-MDT0000 complete [ 3302.377465] LustreError: 103718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.405385] LustreError: 103718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3305.555248] LustreError: 103704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466795 with bad export cookie 5100385787477535019 [ 3305.558492] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.578013] LustreError: 103704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.909338] Lustre: server umount lustre-MDT0001 complete [ 3320.953211] Lustre: server umount lustre-OST0000 complete [ 3323.873077] Lustre: 93286:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466797/real 1788466797] req@ffffa05e93755c00 x1875342736592768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3323.909121] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.074573] Lustre: server umount lustre-OST0001 complete [ 3331.769542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3342.431813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3351.131444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3361.959350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3384.419306] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3396.559528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.585043] Lustre: lustre-MDT0000: reset Object Index mappings [ 3397.437347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.446840] Lustre: Skipped 2 previous similar messages [ 3402.171576] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3410.851947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.884520] Lustre: lustre-MDT0001: reset Object Index mappings [ 3416.736191] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3419.952590] Lustre: 115896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3426.433870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.918715] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3437.228328] LustreError: 116250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.254074] LustreError: 116250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3437.260061] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3443.584296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.712162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3449.369264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3449.372591] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3451.285935] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3458.825796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3462.184706] Lustre: 117773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3513.947918] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:23:21 (1788467001) [ 3539.424356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3539.430896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.436986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.441467] Lustre: Skipped 3 previous similar messages [ 3544.597450] Lustre: server umount lustre-MDT0000 complete [ 3548.237663] LustreError: 114728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467038 with bad export cookie 5100385787477581870 [ 3548.249394] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.807943] Lustre: server umount lustre-MDT0001 complete [ 3565.323776] Lustre: server umount lustre-OST0000 complete [ 3574.927302] Lustre: server umount lustre-OST0001 complete [ 3581.262605] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3588.782264] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3624.843915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.253908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3625.329118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3625.428077] Lustre: lustre-MDT0000: new disk, initializing [ 3625.586444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.589214] Lustre: Skipped 3 previous similar messages [ 3625.605103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.975472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3638.172876] Lustre: server umount lustre-MDT0000 complete [ 3648.273518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.695556] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3648.705150] Lustre: 122038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.713114] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3653.588813] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3658.938715] Lustre: server umount lustre-MDT0000 complete [ 3669.596487] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:25:57 (1788467157) [ 3671.619565] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3674.255277] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:26:01 (1788467161) [ 3694.921833] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3704.703331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.230704] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3718.378867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.615892] Lustre: 124088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3718.712728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3718.723257] Lustre: Skipped 1 previous similar message [ 3718.870520] Lustre: lustre-MDT0001: new disk, initializing [ 3719.093962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3719.111238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3724.583826] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3729.357702] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3736.973891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3737.334928] Lustre: lustre-OST0000: new disk, initializing [ 3737.338263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3737.348108] Lustre: 125666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3739.423195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3739.434134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3739.561254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3743.354191] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3753.076530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.294428] Lustre: lustre-OST0001: new disk, initializing [ 3753.298907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3753.305320] Lustre: 126535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3755.329768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3755.349875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3755.494630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3759.321470] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3768.442631] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3771.878664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4113.085425] Lustre: 132943:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4113.091241] Lustre: 132943:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4113.094587] Lustre: 132943:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4113.098482] Lustre: 132943:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4113.102709] Lustre: 132943:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4113.106466] Lustre: 132943:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4362.415741] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:37:29 (1788467849) [ 4379.618788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4379.624810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.634310] Lustre: Skipped 5 previous similar messages [ 4379.637668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.641389] Lustre: Skipped 8 previous similar messages [ 4383.654723] Lustre: server umount lustre-MDT0000 complete [ 4385.251386] LustreError: 124101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.283404] LustreError: 124101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4386.771456] LustreError: 132958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467876 with bad export cookie 5100385787477649154 [ 4386.778801] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4386.779120] LustreError: 132958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4390.371695] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.380334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4390.381922] Lustre: Skipped 4 previous similar messages [ 4390.396207] Lustre: Skipped 5 previous similar messages [ 4401.119427] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4401.406430] Lustre: server umount lustre-MDT0001 complete [ 4413.747532] Lustre: server umount lustre-OST0000 complete [ 4424.898865] Lustre: server umount lustre-OST0001 complete [ 4433.494258] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4444.179918] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4477.150376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.341736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4477.357935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4477.418407] Lustre: lustre-MDT0000: new disk, initializing [ 4477.471751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.476046] Lustre: Skipped 5 previous similar messages [ 4477.492368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4482.256881] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4491.632395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.756514] Lustre: 135993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4491.827323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4491.829848] Lustre: Skipped 1 previous similar message [ 4491.902179] Lustre: lustre-MDT0001: new disk, initializing [ 4492.053452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.057793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4495.875834] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4503.905583] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.765739] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4514.817129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.014677] Lustre: lustre-OST0000: new disk, initializing [ 4515.019334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4515.027561] Lustre: 137961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.266682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4516.278974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4516.350986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4520.448540] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4530.783728] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4542.602291] Lustre: lustre-MDD0000: changelog on [ 4545.573383] Lustre: lustre-MDD0001: changelog on [ 6700.802355] Lustre: lustre-MDD0001: changelog off [ 6702.867709] Lustre: 135998:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6704.112695] Lustre: lustre-MDD0000: changelog off [ 6713.315297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6713.345804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.362845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6713.371725] Lustre: Skipped 5 previous similar messages [ 6717.064337] Lustre: server umount lustre-OST0000 complete [ 6723.554142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.589073] Lustre: Skipped 1 previous similar message [ 6723.606271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6723.625170] Lustre: Skipped 1 previous similar message [ 6728.208709] Lustre: server umount lustre-MDT0000 complete [ 6728.680228] LustreError: 136003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.732335] LustreError: 136003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6733.807763] LustreError: 136796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6733.837279] LustreError: 136796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6734.031397] LustreError: 135983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470223 with bad export cookie 5100385787478425496 [ 6734.041449] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.044713] LustreError: 135983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6734.401077] Lustre: server umount lustre-MDT0001 complete [ 6749.485472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 6753.484614] Key type lgssc unregistered [ 6753.889880] LNet: 147151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6753.899306] LNetError: 147151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6753.918979] LNet: Removed LNI 192.168.201.110@tcp [ 6755.159168] Key type .llcrypt unregistered [ 6755.160836] Key type ._llcrypt unregistered [ 6779.610935] Key type ._llcrypt registered [ 6779.614621] Key type .llcrypt registered [ 6779.875979] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6802.812595] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6804.463670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6804.476158] alg: No test for adler32 (adler32-zlib) [ 6805.698629] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 6805.956245] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 6807.639178] Key type lgssc registered [ 6809.146865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6855.548203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6855.599102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.948526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6856.997242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6857.106859] Lustre: lustre-MDT0000: new disk, initializing [ 6857.249614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.277393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6862.573230] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6875.900529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6876.035821] Lustre: 150351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6876.108661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6876.110826] Lustre: Skipped 1 previous similar message [ 6876.200577] Lustre: lustre-MDT0001: new disk, initializing [ 6876.329307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.394765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6876.423190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6881.876491] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6892.088830] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6898.861700] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6906.467224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.887715] Lustre: lustre-OST0000: new disk, initializing [ 6906.892889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6906.901602] Lustre: 152323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6907.003211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.095490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6913.101917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6913.210847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6914.589461] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6925.013727] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6930.792526] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_module obdecho/obdecho [ 6933.147519] LustreError: 153326:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6933.159717] LustreError: 153326:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6935.135371] LustreError: 153422:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6949.355346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.357525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6949.376464] Lustre: Skipped 2 previous similar messages [ 6949.390862] Lustre: Skipped 2 previous similar messages [ 6954.465215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.474065] Lustre: Skipped 2 previous similar messages [ 6955.131828] Lustre: server umount lustre-MDT0000 complete [ 6959.665058] LustreError: 150345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470449 with bad export cookie 4803836527631216444 [ 6959.683496] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.219975] Lustre: server umount lustre-MDT0001 complete [ 6975.613739] Lustre: server umount lustre-OST0000 complete [ 7008.719434] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7024.655558] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7065.714916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.048422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7066.097736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7066.200669] Lustre: lustre-MDT0000: new disk, initializing [ 7066.341881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.404233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7072.069741] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7085.025520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.231747] Lustre: 157161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7085.346990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7085.354954] Lustre: Skipped 1 previous similar message [ 7085.488288] Lustre: lustre-MDT0001: new disk, initializing [ 7085.625473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.711885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7085.734915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7092.105202] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7103.027860] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7109.433974] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7117.948928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.419598] Lustre: lustre-OST0000: new disk, initializing [ 7118.425901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7118.433219] Lustre: 159130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7118.562961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.631892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7126.652082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7126.789903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7127.225984] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7138.670463] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.467875] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7145.717318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7151.579697] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7151.867484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7157.731765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7157.732749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.741826] LustreError: Skipped 1 previous similar message [ 7157.742514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.760200] Lustre: Skipped 1 previous similar message [ 7157.777673] Lustre: Skipped 1 previous similar message [ 7160.789689] Lustre: server umount lustre-OST0000 complete [ 7167.459312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7167.478531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.494530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7172.078826] Lustre: server umount lustre-MDT0000 complete [ 7173.095408] LustreError: 158558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.123178] LustreError: 158558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7177.361550] LustreError: 157154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470667 with bad export cookie 4803836527631217403 [ 7177.379564] LustreError: 157154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7177.399130] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.669331] Lustre: server umount lustre-MDT0001 complete [ 7182.742491] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:24:29 (1788470669) [ 7184.745891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7187.539572] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:24:34 (1788470674) [ 7197.687057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.759112] LustreError: 161542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.930349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.321411] LustreError: 161543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.587829] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7208.504905] LustreError: 161542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.544392] LustreError: 161543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.226843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.334707] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7237.001054] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.760531] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7253.522472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.151794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.158051] Lustre: Skipped 1 previous similar message [ 7262.465360] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7272.747380] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7283.169759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.173754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.181649] Lustre: Skipped 1 previous similar message [ 7283.187586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.195107] Lustre: Skipped 1 previous similar message [ 7285.047633] Lustre: server umount lustre-MDT0000 complete [ 7289.096810] LustreError: 162594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.149349] LustreError: 162594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7289.676932] LustreError: 161525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470779 with bad export cookie 4803836527631218285 [ 7289.682120] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.696579] LustreError: 161525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7294.257975] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 7294.267443] Lustre: Skipped 4 previous similar messages [ 7294.436757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.461682] Lustre: Skipped 2 previous similar messages [ 7296.200895] Lustre: server umount lustre-MDT0001 complete [ 7304.829190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7322.020179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.346699] LustreError: 165389:0:(llog.c:1655:llog_backup()) MGC192.168.201.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7322.351956] Lustre: 165389:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7323.970663] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.993612] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7360.771796] LustreError: 165395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7360.815362] LustreError: 165395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 7383.007906] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7383.030945] Lustre: 165389:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.110@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7424.029309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7425.306869] LustreError: 165395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.330208] LustreError: 165395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7429.104892] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7430.761335] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7442.236503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.592579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7466.513103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7467.805033] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7477.570383] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7479.896125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7488.243624] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7490.632537] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7512.173882] Lustre: server umount lustre-MDT0000 complete [ 7512.546778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.546991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7512.561431] Lustre: Skipped 1 previous similar message [ 7515.932855] LustreError: 161524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471005 with bad export cookie 4803836527631224774 [ 7515.946704] LustreError: 161524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7515.948337] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7516.182518] Lustre: server umount lustre-MDT0001 complete [ 7530.298712] Lustre: server umount lustre-OST0000 complete [ 7557.736116] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 7570.428886] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7614.984444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.281460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7615.317908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7615.395665] Lustre: lustre-MDT0000: new disk, initializing [ 7615.460701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.473911] Lustre: Skipped 1 previous similar message [ 7615.486476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7621.338714] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7632.745234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.890466] Lustre: 170501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7633.093753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7638.549853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7649.716041] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7656.774567] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7665.449465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7665.904464] Lustre: lustre-OST0000: new disk, initializing [ 7665.909320] Lustre: Skipped 1 previous similar message [ 7665.922355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7665.929281] Lustre: Skipped 2 previous similar messages [ 7665.937958] Lustre: 172471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7673.485351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7673.505560] Lustre: Skipped 1 previous similar message [ 7673.508734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7673.674737] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7673.708140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7684.823712] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7691.043781] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7691.344873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7697.216466] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7697.427214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7704.544440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7704.554691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.582638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7704.589944] Lustre: Skipped 2 previous similar messages [ 7714.271189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7714.542718] Lustre: server umount lustre-OST0000 complete [ 7725.861286] Lustre: server umount lustre-MDT0000 complete [ 7730.149386] LustreError: 170514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7730.175327] LustreError: 170514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 7730.529295] LustreError: 170493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471220 with bad export cookie 4803836527631241574 [ 7730.534548] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.547183] LustreError: 170493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7731.006911] Lustre: server umount lustre-MDT0001 complete [ 7741.598950] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:33:48 (1788471228) [ 7749.834090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.379585] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7765.788243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.549687] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7780.722651] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7788.631867] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7799.017946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.495873] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7820.933874] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.455738] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:35:19 (1788471319) [ 7841.763189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7841.765814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.772305] LustreError: Skipped 1 previous similar message [ 7841.775308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7841.775317] Lustre: Skipped 6 previous similar messages [ 7841.835716] Lustre: Skipped 4 previous similar messages [ 7844.387201] Lustre: server umount lustre-OST0000 complete [ 7854.837559] Lustre: server umount lustre-MDT0000 complete [ 7859.084667] LustreError: 174865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471348 with bad export cookie 4803836527631242456 [ 7859.094070] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7859.101884] LustreError: 174865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7859.483977] Lustre: server umount lustre-MDT0001 complete [ 7872.284648] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 7875.204905] Key type lgssc unregistered [ 7875.543977] LNet: 178616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7875.560176] LNetError: 178616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7875.587166] LNet: Removed LNI 192.168.201.110@tcp [ 7876.596185] Key type .llcrypt unregistered [ 7876.602325] Key type ._llcrypt unregistered [ 7897.289297] Key type ._llcrypt registered [ 7897.292778] Key type .llcrypt registered [ 7897.442940] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 7898.464861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7898.544062] alg: No test for adler32 (adler32-zlib) [ 7899.654049] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 7899.915308] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 7901.583820] Key type lgssc registered [ 7902.965663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7912.220406] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7912.237611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.776699] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7932.895319] LustreError: 179859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 7938.015605] LustreError: 179859:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7938.048285] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7938.070719] LustreError: MGC192.168.201.110@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7938.106720] LustreError: 179859:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7938.131405] LustreError: 179859:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7938.159452] LustreError: 179859:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7938.302377] Lustre: server umount lustre-OST0000 complete [ 7938.304337] LustreError: 179859:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7947.517244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.995810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.980915] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7958.501782] LustreError: 180358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.525571] LustreError: 180358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7963.617365] LustreError: 180357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.143664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.616839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7971.420570] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 7979.375918] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7985.796196] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7994.650437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7994.858568] LustreError: 182173:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7994.870515] Lustre: 182173:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.110@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7995.016676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.001291] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8012.994901] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8016.864683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8016.871143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8016.897985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.961139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.970264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.996032] Lustre: Skipped 1 previous similar message [ 8022.483509] Lustre: server umount lustre-MDT0000 complete [ 8026.092902] LustreError: 180357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.130410] LustreError: 180357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8027.044172] LustreError: 180339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471516 with bad export cookie 209469643131946656 [ 8027.052452] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.057384] LustreError: 180339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8027.523776] Lustre: server umount lustre-MDT0001 complete [ 8043.534730] Lustre: server umount lustre-OST0000 complete [ 8055.389410] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8070.969766] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8111.189950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.429340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8111.449348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8111.562773] Lustre: lustre-MDT0000: new disk, initializing [ 8111.651167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.673739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8117.402818] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8129.755320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.897826] Lustre: 185723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8129.947323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8129.957161] Lustre: Skipped 1 previous similar message [ 8130.034715] Lustre: lustre-MDT0001: new disk, initializing [ 8130.130885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8130.165845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8130.176432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8135.391967] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8146.669493] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8154.508466] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8162.636167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.175861] Lustre: lustre-OST0000: new disk, initializing [ 8163.182637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.200114] Lustre: 187698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.360991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.477447] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8172.654862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8172.682194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8172.926845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8183.383401] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8188.749628] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8188.975413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8193.889857] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8194.178291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8198.633703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8198.636096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.646437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8198.646445] Lustre: Skipped 2 previous similar messages [ 8198.649042] LustreError: Skipped 1 previous similar message [ 8198.685195] Lustre: Skipped 1 previous similar message [ 8203.755429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8203.765935] Lustre: Skipped 1 previous similar message [ 8208.868796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.875941] Lustre: Skipped 2 previous similar messages [ 8210.399265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8210.780220] Lustre: server umount lustre-OST0000 complete [ 8217.056185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8217.062814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.097696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.552727] Lustre: server umount lustre-MDT0000 complete [ 8224.226692] LustreError: 186842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8224.272110] LustreError: 186842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8224.948605] LustreError: 185718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471714 with bad export cookie 209469643131947363 [ 8224.963954] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8224.968209] LustreError: 185718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8225.254624] Lustre: server umount lustre-MDT0001 complete [ 8236.981419] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:42:04 (1788471724) [ 8260.499901] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 8264.144807] Key type lgssc unregistered [ 8264.430863] LNet: 190644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.438364] LNetError: 190644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.455639] LNet: Removed LNI 192.168.201.110@tcp [ 8265.453913] Key type .llcrypt unregistered [ 8265.458711] Key type ._llcrypt unregistered [ 8284.901603] Key type ._llcrypt registered [ 8284.903742] Key type .llcrypt registered [ 8285.030933] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8286.429632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8286.450120] alg: No test for adler32 (adler32-zlib) [ 8287.449934] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 8287.695525] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 8289.415233] Key type lgssc registered [ 8290.500203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8298.933989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8298.956339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.505096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.763743] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8310.755048] LustreError: 191861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.771864] LustreError: 191861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8314.691668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.995049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8319.249423] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8326.463553] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.295126] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8340.988210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8341.171695] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8341.174437] LustreError: 193722:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8341.179132] Lustre: 193722:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.110@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8341.283564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.579978] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8356.284628] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8359.785820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.808302] Lustre: Skipped 1 previous similar message [ 8359.810667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.025557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.039471] Lustre: Skipped 4 previous similar messages [ 8370.144975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.148567] Lustre: Skipped 1 previous similar message [ 8373.215379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8373.361721] Lustre: server umount lustre-MDT0000 complete [ 8375.271474] LustreError: 192588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.292720] LustreError: 192588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8378.336162] LustreError: 191842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471868 with bad export cookie 1425629458244532841 [ 8378.345664] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.363125] LustreError: 191842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8380.319471] LustreError: 192590:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a9e3fa44 x1875348582069376/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8380.387374] LustreError: 191855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.418497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.448874] Lustre: Skipped 1 previous similar message [ 8380.842934] Lustre: server umount lustre-MDT0001 complete [ 8386.363479] Lustre: server umount lustre-OST0000 complete [ 8397.815917] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8409.532341] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8450.574485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.874470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8450.924361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8451.053268] Lustre: lustre-MDT0000: new disk, initializing [ 8451.162363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.198389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8455.160216] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8466.540843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.729404] Lustre: 197250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8466.802555] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8466.805033] Lustre: Skipped 1 previous similar message [ 8466.812545] LustreError: 198033:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8466.820895] LustreError: 198033:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8466.829864] Lustre: 198033:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8466.873553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8466.888513] Lustre: Skipped 1 previous similar message [ 8467.028316] Lustre: lustre-MDT0001: new disk, initializing [ 8467.179517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.241928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8467.269334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8472.419445] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8482.531678] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8487.956885] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8494.291517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8494.534095] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8494.536390] Lustre: Skipped 1 previous similar message [ 8494.538646] LustreError: 199195:0:(llog.c:1696:llog_backup()) MGC192.168.201.110@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8494.546952] LustreError: 199195:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8494.551827] Lustre: 199195:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.110@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8494.559594] Lustre: 199195:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8494.582390] Lustre: lustre-OST0000: new disk, initializing [ 8494.590200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8494.597558] Lustre: 199219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8494.643838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8500.293890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8500.304876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8500.435529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8501.743497] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8511.986874] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8517.283544] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8517.543517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8523.905918] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8524.214370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8526.310978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.320724] Lustre: Skipped 1 previous similar message [ 8526.322892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.324780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8526.335557] LustreError: Skipped 1 previous similar message [ 8526.351670] Lustre: Skipped 1 previous similar message [ 8531.428242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8531.441659] Lustre: Skipped 1 previous similar message [ 8532.729212] Lustre: server umount lustre-OST0000 complete [ 8539.110875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8539.124193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.674735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8541.693681] Lustre: Skipped 2 previous similar messages [ 8543.738965] Lustre: server umount lustre-MDT0000 complete [ 8546.787127] LustreError: 199890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.806082] LustreError: 199890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8547.983182] LustreError: 197241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472037 with bad export cookie 1425629458244533562 [ 8547.985783] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8548.005030] LustreError: 197241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8548.468816] Lustre: server umount lustre-MDT0001 complete [ 8557.989495] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:47:25 (1788472045) [ 8576.687678] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8589.067222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.455041] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.523307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.395783] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8600.040259] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.075699] LustreError: 202348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8605.755584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.104214] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8614.312739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8621.802861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.196539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.201592] Lustre: Skipped 1 previous similar message [ 8629.422296] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8638.974983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.155198] Lustre: lustre-OST0001: new disk, initializing [ 8639.164433] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8639.171803] Lustre: 204656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8647.716589] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8648.896276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8648.910080] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8649.139148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8660.586128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8668.091934] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8670.534764] Lustre: DEBUG MARKER: ADD OST3 [ 8676.571195] loop: module loaded [ 8676.680061] loop0: detected capacity change from 0 to 204800000 [ 8677.028621] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8677.048634] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8677.076166] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8685.272366] Lustre: DEBUG MARKER: STOP OST3 [ 8685.383755] loop0: detected capacity change from 0 to 204800000 [ 8692.707600] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.011223] Lustre: lustre-OST0003: new disk, initializing [ 8693.017124] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8693.032172] Lustre: 206728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8693.197098] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8693.204135] Lustre: Skipped 1 previous similar message [ 8694.643216] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8694.870144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8694.883799] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8694.937771] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8696.360143] Lustre: Failing over lustre-OST0003 [ 8696.573309] Lustre: server umount lustre-OST0003 complete [ 8696.672790] LustreError: 206832:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8699.907382] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.927336] Lustre: Skipped 1 previous similar message [ 8699.936761] LustreError: 205072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.936945] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8700.014555] LustreError: 205072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8701.593824] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8710.706404] Lustre: DEBUG MARKER: START OST3 again [ 8720.812345] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8721.452982] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8723.258233] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8723.381182] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8723.381632] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8729.908621] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8752.632335] Lustre: Failing over lustre-OST0003 [ 8752.756487] Lustre: server umount lustre-OST0003 complete [ 8754.149508] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8754.160864] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.188560] LustreError: 203833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.212997] LustreError: 203833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 8772.582634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.584376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8772.603592] Lustre: Skipped 1 previous similar message [ 8772.612583] Lustre: Skipped 3 previous similar messages [ 8777.267901] Lustre: server umount lustre-MDT0000 complete [ 8781.626645] LustreError: 202328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472271 with bad export cookie 1425629458244534444 [ 8781.647536] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8781.654572] LustreError: 202328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8788.267980] Lustre: server umount lustre-MDT0001 complete [ 8793.574411] Lustre: server umount lustre-OST0000 complete [ 8797.920088] Lustre: server umount lustre-OST0001 complete [ 8806.399189] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 8815.756486] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 8860.138954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.475427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8860.498406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8860.576152] Lustre: lustre-MDT0000: new disk, initializing [ 8860.623635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.632571] Lustre: Skipped 1 previous similar message [ 8860.644337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8865.976522] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8877.146595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8877.329487] Lustre: 212119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8877.472950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8877.476909] Lustre: Skipped 1 previous similar message [ 8877.580663] Lustre: lustre-MDT0001: new disk, initializing [ 8877.708735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8877.721667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8881.259446] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8892.613390] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8899.215566] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8907.727771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.376393] Lustre: 214090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8909.998588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8910.011256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8910.298568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8918.220616] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 8930.828406] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8938.405838] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8938.928235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8945.496624] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8946.009288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8951.266906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.271807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8951.284257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8951.284266] Lustre: Skipped 6 previous similar messages [ 8951.286078] Lustre: Skipped 4 previous similar messages [ 8951.374185] LustreError: Skipped 2 previous similar messages [ 8955.119610] Lustre: server umount lustre-OST0000 complete [ 8974.303193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8974.640157] Lustre: server umount lustre-MDT0000 complete [ 8976.870606] LustreError: 212132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.898573] LustreError: 212132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8979.311225] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472469 with bad export cookie 1425629458244536740 [ 8979.324388] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8979.326550] LustreError: 212113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8980.099033] Lustre: server umount lustre-MDT0001 complete [ 8984.433171] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:54:31 (1788472471) [ 9002.648330] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9015.000916] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9057.693592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.019496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9058.089611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9058.094512] Lustre: Skipped 1 previous similar message [ 9058.203935] Lustre: lustre-MDT0000: new disk, initializing [ 9058.211301] Lustre: Skipped 1 previous similar message [ 9058.355051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9063.931793] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9076.154333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.365989] Lustre: 218703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9076.658226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9082.105472] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9093.013395] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9098.914714] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9106.569953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9106.809279] Lustre: 220678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9108.453148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9113.838786] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9124.247460] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9129.486424] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9129.706539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9135.506492] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9135.721692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9139.170472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9139.172547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.184907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9139.193042] Lustre: Skipped 3 previous similar messages [ 9139.208432] Lustre: Skipped 8 previous similar messages [ 9144.117988] Lustre: server umount lustre-OST0000 complete [ 9154.846589] Lustre: server umount lustre-MDT0000 complete [ 9158.883646] LustreError: 219502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472648 with bad export cookie 1425629458244537622 [ 9158.896286] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9159.472099] Lustre: server umount lustre-MDT0001 complete [ 9169.202370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.747701] LustreError: 222894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.865370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.870716] Lustre: Skipped 5 previous similar messages [ 9175.575494] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9185.351770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.184259] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9201.420575] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9208.862689] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9217.729973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9226.168395] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9235.071964] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9251.302670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9257.140624] Lustre: server umount lustre-OST0000 complete [ 9267.537890] Lustre: server umount lustre-MDT0000 complete [ 9271.776273] LustreError: 222874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472761 with bad export cookie 1425629458244538504 [ 9271.791506] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.793617] LustreError: 222874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9272.161539] Lustre: server umount lustre-MDT0001 complete [ 9373.700248] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 9376.200829] Key type lgssc unregistered [ 9376.547630] LNet: 226597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9376.554822] LNetError: 226597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9376.580521] LNet: Removed LNI 192.168.201.110@tcp [ 9377.358222] Key type .llcrypt unregistered [ 9377.360765] Key type ._llcrypt unregistered [ 9403.772273] Key type ._llcrypt registered [ 9403.778080] Key type .llcrypt registered [ 9403.893073] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9422.180503] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9423.316757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9423.366580] alg: No test for adler32 (adler32-zlib) [ 9424.557706] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 9424.809096] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 9426.439117] Key type lgssc registered [ 9427.737956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9471.098174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9471.127126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.509809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9472.553611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9472.642455] Lustre: lustre-MDT0000: new disk, initializing [ 9472.732339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9472.751928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9478.318895] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9489.856996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.988708] Lustre: 229960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9490.036946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9490.039763] Lustre: Skipped 1 previous similar message [ 9490.134473] Lustre: lustre-MDT0001: new disk, initializing [ 9490.247911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9490.295490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9490.309300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9495.617493] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9505.142351] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9512.133622] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9520.014353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.371409] Lustre: lustre-OST0000: new disk, initializing [ 9520.374720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9520.379473] Lustre: 231927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.454940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.288838] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9527.409968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9527.422868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9527.520611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9537.980934] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9543.368460] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9543.628969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9548.699566] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9548.996296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9553.376070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9553.379141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.392158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9553.406725] Lustre: Skipped 1 previous similar message [ 9558.500618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.505048] Lustre: Skipped 2 previous similar messages [ 9563.625603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9565.152169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9565.391537] Lustre: server umount lustre-OST0000 complete [ 9572.323283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.340477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.355245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.372965] Lustre: Skipped 1 previous similar message [ 9573.861819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.037369] Lustre: server umount lustre-MDT0000 complete [ 9581.004338] LustreError: 229953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473070 with bad export cookie 16004115493554823078 [ 9581.020814] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9581.021340] LustreError: 229953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9581.527507] Lustre: server umount lustre-MDT0001 complete [ 9604.925770] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9619.308515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.752907] LustreError: 234867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9619.773036] LustreError: 234867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9619.863495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.069209] LustreError: 234868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.497626] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9629.219690] LustreError: 234867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.280391] LustreError: 234868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.066420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.376872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9642.169573] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9645.636334] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9652.974242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9653.320771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9660.471960] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9668.532439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9668.792421] Lustre: lustre-OST0001: new disk, initializing [ 9668.796105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9668.803391] Lustre: 237178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9675.937599] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9676.947677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9676.966383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9677.171047] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9689.687886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9699.611610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9714.707388] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:06:41 (1788473201) [ 9718.245765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.248172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9718.251883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.251893] Lustre: Skipped 2 previous similar messages [ 9718.270503] Lustre: Skipped 1 previous similar message [ 9723.229081] Lustre: server umount lustre-MDT0000 complete [ 9723.366472] LustreError: 235811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9723.383665] LustreError: 235811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9727.274397] LustreError: 237177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473217 with bad export cookie 16004115493554823960 [ 9727.284518] LustreError: 237177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9727.292150] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.745312] Lustre: server umount lustre-MDT0001 complete [ 9743.213565] Lustre: server umount lustre-OST0000 complete [ 9744.671440] Lustre: 228182:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788473218/real 1788473218] req@ffffa05e935f1c00 x1875349774388992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788473234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9744.731255] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.777733] Lustre: Skipped 2 previous similar messages [ 9748.453780] Lustre: 228185:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788473222/real 1788473222] req@ffffa05f875b4a80 x1875349774389248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788473238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9749.072384] Lustre: server umount lustre-OST0001 complete [ 9757.866203] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 9769.689446] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 9813.181982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.534727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9813.562735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9813.644695] Lustre: lustre-MDT0000: new disk, initializing [ 9813.710500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9813.721183] Lustre: Skipped 1 previous similar message [ 9813.769301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9819.859123] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9831.352296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.601572] Lustre: 241122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9831.711216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9831.715446] Lustre: Skipped 1 previous similar message [ 9831.865312] Lustre: lustre-MDT0001: new disk, initializing [ 9832.076652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9832.095765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9839.002512] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9849.435114] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9855.039886] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9863.613635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9863.975238] Lustre: lustre-OST0000: new disk, initializing [ 9863.984188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9864.003565] Lustre: 243090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9864.142577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.153847] Lustre: Skipped 1 previous similar message [ 9865.589716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9865.605761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9865.719297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9870.829746] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9881.332258] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9885.967616] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9886.173956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9891.005545] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9891.349349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9896.428399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.432962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.438668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.438677] Lustre: Skipped 3 previous similar messages [ 9896.475331] Lustre: Skipped 1 previous similar message [ 9899.700426] Lustre: server umount lustre-OST0000 complete [ 9903.585951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9903.595180] LustreError: Skipped 1 previous similar message [ 9909.626367] Lustre: server umount lustre-MDT0000 complete [ 9911.782522] LustreError: 241133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.823575] LustreError: 241133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9913.807970] LustreError: 241116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473403 with bad export cookie 16004115493554826095 [ 9913.824975] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.825611] LustreError: 241116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9914.173453] Lustre: server umount lustre-MDT0001 complete [ 9924.664969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.813765] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9930.213715] LustreError: 245306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9930.243440] LustreError: 245306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9939.310292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.717632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9939.734478] Lustre: Skipped 1 previous similar message [ 9944.905182] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9954.288400] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9961.791846] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9970.207843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.943639] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 9986.704652] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10138.592697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10138.599074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.609491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10138.609499] Lustre: Skipped 4 previous similar messages [10138.643804] Lustre: Skipped 3 previous similar messages [10140.975054] Lustre: server umount lustre-OST0000 complete [10159.072839] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10159.229547] Lustre: server umount lustre-MDT0000 complete [10163.521267] LustreError: 245287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473653 with bad export cookie 16004115493554826977 [10163.535024] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10164.121039] Lustre: server umount lustre-MDT0001 complete [10255.403279] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [10258.552626] Key type lgssc unregistered [10258.912564] LNet: 248917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10258.917801] LNetError: 248917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10258.938188] LNet: Removed LNI 192.168.201.110@tcp [10260.006181] Key type .llcrypt unregistered [10260.008787] Key type ._llcrypt unregistered [10291.895695] Key type ._llcrypt registered [10291.897659] Key type .llcrypt registered [10292.195903] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [10314.227258] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10315.899770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10315.989554] alg: No test for adler32 (adler32-zlib) [10317.255183] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [10317.711748] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [10319.423525] Key type lgssc registered [10321.027413] Lustre: Echo OBD driver; http://www.lustre.org/ [10375.246589] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10386.630221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10386.644878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10387.957211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10387.982673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10388.075115] Lustre: lustre-MDT0000: new disk, initializing [10388.199535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10388.222417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10393.145928] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10405.531476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.720466] Lustre: 252988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10405.802795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10405.805343] Lustre: Skipped 1 previous similar message [10405.894462] Lustre: lustre-MDT0001: new disk, initializing [10406.042333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10406.095666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10406.111758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10412.993994] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10418.458643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10425.101533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10425.451988] Lustre: lustre-OST0000: new disk, initializing [10425.455570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10425.460858] Lustre: 254620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10425.535910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10429.993745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10429.999590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10430.079780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10431.581872] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10442.228081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10442.385311] Lustre: lustre-OST0001: new disk, initializing [10442.390461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10442.396298] Lustre: 255491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10442.509164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10450.121470] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10450.719454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10450.732062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10450.877147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10462.767945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10470.340889] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10486.760403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10486.763148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10486.777523] Lustre: Skipped 3 previous similar messages [10486.785051] Lustre: Skipped 2 previous similar messages [10490.427302] Lustre: server umount lustre-MDT0000 complete [10494.167814] LustreError: 256315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473983 with bad export cookie 6916999624783698307 [10494.184018] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.189991] LustreError: 256315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10494.556419] Lustre: server umount lustre-MDT0001 complete [10508.705814] Lustre: server umount lustre-OST0000 complete [10521.610352] Lustre: server umount lustre-OST0001 complete [10533.857106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10542.880289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10550.085372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10560.895626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10585.822969] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10599.026032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.075568] Lustre: lustre-MDT0000: reset Object Index mappings [10599.841798] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.878035] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10600.006321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10605.030946] LustreError: 260502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.732463] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10610.160950] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.278171] LustreError: 260502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.862084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10615.890729] Lustre: lustre-MDT0001: reset Object Index mappings [10616.470053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10621.331272] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10624.240507] Lustre: 261646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10631.140482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10638.387511] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10641.841807] LustreError: 262001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10644.919331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10649.262834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.773477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10649.776475] Lustre: Skipped 1 previous similar message [10655.260234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10659.648903] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10669.773624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10673.994897] Lustre: 263519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10680.808851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10680.819449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10680.831147] Lustre: Skipped 1 previous similar message [10680.846738] Lustre: Skipped 2 previous similar messages [10681.586408] Lustre: server umount lustre-MDT0000 complete [10685.315807] LustreError: 260483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474175 with bad export cookie 6916999624783705097 [10685.320842] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10685.324796] LustreError: 260483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10685.611931] Lustre: server umount lustre-MDT0001 complete [10700.026346] Lustre: server umount lustre-OST0000 complete [10701.279282] Lustre: 250491:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474175/real 1788474175] req@ffffa05fbfbeb100 x1875350710779904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788474191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10701.305279] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10701.322474] Lustre: Skipped 2 previous similar messages [10703.748384] Lustre: server umount lustre-OST0001 complete [10723.736391] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:23:30 (1788474210) [10743.436653] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [10755.849274] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10799.367249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.654661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10799.697910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10799.892716] Lustre: lustre-MDT0000: new disk, initializing [10800.101827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10800.145968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10806.627478] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10818.659530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.857619] Lustre: 267773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10818.939952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10818.945267] Lustre: Skipped 1 previous similar message [10819.045117] Lustre: lustre-MDT0001: new disk, initializing [10819.208650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10819.219499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10824.027870] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10835.681816] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10843.335278] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10851.919118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10852.446593] Lustre: lustre-OST0000: new disk, initializing [10852.452105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10852.464300] Lustre: 269744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10854.146858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10854.172710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10854.405075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10860.766803] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10872.277802] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10879.295591] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10879.548137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10884.770898] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10885.195487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10890.212637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10890.217446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.221405] LustreError: Skipped 1 previous similar message [10890.222282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10890.222288] Lustre: Skipped 1 previous similar message [10890.253559] Lustre: Skipped 1 previous similar message [10893.728042] Lustre: server umount lustre-OST0000 complete [10900.454843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10900.474338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10900.483943] Lustre: Skipped 1 previous similar message [10905.572339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10905.577744] Lustre: Skipped 1 previous similar message [10912.223161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10912.635675] Lustre: server umount lustre-MDT0000 complete [10915.842764] LustreError: 267779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.888180] LustreError: 267779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10918.621194] LustreError: 267765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474408 with bad export cookie 6916999624783707190 [10918.622462] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.647038] LustreError: 267765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10919.415445] Lustre: server umount lustre-MDT0001 complete [10943.503484] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [10957.006116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.558974] LustreError: 272682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.706847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.713389] Lustre: Skipped 2 previous similar messages [10962.957266] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10974.787908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.037882] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [10986.460390] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10993.690564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11001.254406] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11011.275768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11011.531407] Lustre: lustre-OST0001: new disk, initializing [11011.534935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11011.540842] Lustre: 274992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11013.406328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11013.416021] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11013.552149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11018.767931] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11029.042575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11033.314504] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11038.930437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11039.096346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11046.881417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11046.889289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11046.907515] Lustre: Skipped 1 previous similar message [11046.913618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11046.916841] Lustre: Skipped 2 previous similar messages [11053.058332] Lustre: server umount lustre-MDT0000 complete [11054.567587] LustreError: 272683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.594385] LustreError: 272683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11057.769416] LustreError: 272663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474547 with bad export cookie 6916999624783708072 [11057.799978] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11057.805847] LustreError: 272663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11058.555532] Lustre: server umount lustre-MDT0001 complete [11073.395941] Lustre: server umount lustre-OST0000 complete [11076.064534] Lustre: 250489:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474549/real 1788474549] req@ffffa05e88c7e300 x1875350710893440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788474565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11076.145483] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.188090] Lustre: Skipped 3 previous similar messages [11078.242245] Lustre: 250489:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474552/real 1788474552] req@ffffa05fa7aee300 x1875350710893696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788474568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11078.793945] Lustre: server umount lustre-OST0001 complete [11097.640523] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11111.687254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.224490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11112.228645] Lustre: Skipped 3 previous similar messages [11117.313635] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11121.636268] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.671777] LustreError: 278046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11127.406599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.844324] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11136.626420] Lustre: 279187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11146.735782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.234426] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11156.634886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11165.818629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11171.328520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11174.503874] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11185.311411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11190.072110] Lustre: 281065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11198.651644] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11198.774118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11212.123534] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:31:39 (1788474699) [11230.186139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11230.193940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11230.229426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11230.240759] Lustre: Skipped 4 previous similar messages [11235.205063] Lustre: server umount lustre-MDT0000 complete [11239.805789] LustreError: 279566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474729 with bad export cookie 6916999624783714617 [11239.845803] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11250.145133] LustreError: 278042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11250.156480] LustreError: 278042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11254.240506] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11254.777136] Lustre: server umount lustre-MDT0001 complete [11266.849254] Lustre: server umount lustre-OST0000 complete [11271.715554] Lustre: server umount lustre-OST0001 complete [11278.680093] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [11292.448960] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11346.077396] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11358.251204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.488721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11358.512148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11358.561627] Lustre: lustre-MDT0000: new disk, initializing [11358.634343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11363.712294] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11375.671752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.744183] Lustre: 285309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11375.905238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11375.914092] Lustre: Skipped 4 previous similar messages [11375.948491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11383.212729] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11395.805301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.047613] Lustre: 286943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11398.126181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11402.314171] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11411.625517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11411.773851] Lustre: 287812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11413.169961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11413.178114] Lustre: Skipped 1 previous similar message [11413.254840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11418.684902] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11428.772377] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11432.892794] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11432.896260] Lustre: Skipped 1 previous similar message [11451.683810] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:35:39 (1788474939) [11454.434318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.441346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11454.446260] Lustre: Skipped 7 previous similar messages [11454.470460] Lustre: Skipped 13 previous similar messages [11456.670538] Lustre: server umount lustre-MDT0000 complete [11461.192846] LustreError: 285301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474950 with bad export cookie 6916999624783722268 [11461.203089] LustreError: 285301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11461.205670] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11461.649314] Lustre: server umount lustre-MDT0001 complete [11477.354065] Lustre: server umount lustre-OST0000 complete [11481.186278] Lustre: 250491:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788474954/real 1788474954] req@ffffa05f890c0380 x1875350711054976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788474970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11481.857636] Lustre: server umount lustre-OST0001 complete [11492.801890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.445896] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11508.732209] LustreError: 290063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.758344] LustreError: 290063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11509.496596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.259718] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11524.475069] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11531.388411] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11539.490970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11548.042156] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11557.510698] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11585.200927] loop1: detected capacity change from 0 to 204800000 [11585.456401] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.481782] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.511394] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11591.910445] loop1: detected capacity change from 0 to 204800000 [11597.500071] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11597.635194] LustreError: 290051:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1348: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11597.645705] LustreError: 290051:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1348 [11597.650395] LustreError: 290051:0:(mgs_handler.c:517:mgs_target_reg()) fs1348-MDT0001: Error checking target index: rc = -13 [11597.661180] LustreError: fs1348-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11597.667334] LustreError: 293804:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11597.671936] LustreError: 293804:0:(tgt_mount.c:2132:server_put_super()) no obd fs1348-MDT0001 [11597.675400] LustreError: 293804:0:(tgt_mount.c:116:server_deregister_mount()) fs1348-MDT0001 not registered [11597.706257] Lustre: server umount fs1348-MDT0001 complete [11597.709454] LustreError: 293804:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11608.214459] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.407719] Lustre: Setting parameter fs1348-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1348-MDT0001 [11608.486493] Lustre: srv-fs1348-MDT0001: No data found on store. Initialize space. [11608.492332] Lustre: Skipped 4 previous similar messages [11608.661860] Lustre: fs1348-MDT0001: new disk, initializing [11608.667138] Lustre: Skipped 3 previous similar messages [11608.932663] Lustre: srv-fs1348-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11608.943441] Lustre: 294270:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1348-MDT0001-mdtlov: can't create new llog: rc = -115 [11608.951726] LustreError: 294270:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11614.159822] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11621.915396] Lustre: server umount fs1348-MDT0001 complete [11635.179694] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11637.039996] loop1: detected capacity change from 0 to 204800000 [11640.319856] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11644.538217] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.025395] Lustre: 295839:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1348-MDT0001-mdtlov: can't create new llog: rc = -115 [11645.036097] LustreError: 295839:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11650.509424] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11655.594051] Lustre: server umount fs1348-MDT0001 complete [11668.968776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11668.980153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.995300] Lustre: Skipped 2 previous similar messages [11669.002738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11681.248484] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11681.639321] Lustre: server umount lustre-MDT0000 complete [11692.055029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.262289] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11697.121584] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11697.657037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11705.178048] loop1: detected capacity change from 0 to 204800000 [11712.796712] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.048925] Lustre: 298287:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1348-MDT0001-mdtlov: can't create new llog: rc = -115 [11713.061676] LustreError: 298287:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1348-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11723.716863] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11731.796336] Lustre: server umount fs1348-MDT0001 complete [11759.098319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11764.665818] Lustre: server umount lustre-OST0000 complete [11776.135070] Lustre: server umount lustre-MDT0000 complete [11780.900554] LustreError: 290043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475270 with bad export cookie 6916999624783724949 [11780.912555] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11780.912667] LustreError: 290043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11781.271951] Lustre: server umount lustre-MDT0001 complete [11797.322920] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [11801.922555] Key type lgssc unregistered [11802.333786] LNet: 300633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11802.345909] LNetError: 300633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11802.374724] LNet: Removed LNI 192.168.201.110@tcp [11803.360292] Key type .llcrypt unregistered [11803.362982] Key type ._llcrypt unregistered [11828.544329] Key type ._llcrypt registered [11828.549357] Key type .llcrypt registered [11828.782138] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [11829.939966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11829.985802] alg: No test for adler32 (adler32-zlib) [11831.127325] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [11831.349319] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [11833.127205] Key type lgssc registered [11834.036130] Lustre: Echo OBD driver; http://www.lustre.org/ [11843.200376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11843.249375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.025048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11852.234607] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11854.372105] LustreError: 301906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.408279] LustreError: 301906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11859.442789] LustreError: 301905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.552072] LustreError: 301906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.690633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.206734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11870.873735] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11880.102854] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11887.016493] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11896.954891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11897.693280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11906.502416] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11910.001193] LustreError: 303733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.125795] LustreError: 303733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.160463] LustreError: 303733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11916.303868] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.223830] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.274833] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11938.649237] loop1: detected capacity change from 0 to 204800000 [11938.872910] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11938.925208] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11938.944374] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11940.836955] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.852479] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11945.802949] loop1: detected capacity change from 0 to 204800000 [11953.402218] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.682614] Lustre: fs26353-OST0000: new disk, initializing [11953.689269] Lustre: srv-fs26353-OST0000: No data found on store. Initialize space. [11953.705543] Lustre: 305642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11953.854699] Lustre: fs26353-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.627771] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [11974.048712] Lustre: server umount fs26353-OST0000 complete [11974.415189] LustreError: 304359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11974.452825] LustreError: 304359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11989.555983] loop1: detected capacity change from 0 to 204800000 [11992.047294] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11996.503474] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.700968] Lustre: fs26353-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12004.970958] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12016.271041] loop2: detected capacity change from 0 to 204800000 [12016.585179] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12016.612436] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12016.643137] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.474701] loop2: detected capacity change from 0 to 204800000 [12029.819981] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12029.917487] LustreError: fs26353-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12029.945429] LustreError: fs26353-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12029.972399] LustreError: 308685:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [12029.989540] LustreError: 308685:0:(tgt_mount.c:2132:server_put_super()) no obd fs26353-OST0001 [12030.003284] LustreError: 308685:0:(tgt_mount.c:116:server_deregister_mount()) fs26353-OST0001 not registered [12030.080399] Lustre: server umount fs26353-OST0001 complete [12030.085966] LustreError: 308685:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12040.981879] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.013511] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12041.185398] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.465646] Lustre: fs26353-OST0001: new disk, initializing [12041.470151] Lustre: srv-fs26353-OST0001: No data found on store. Initialize space. [12041.480106] Lustre: 309236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12041.616283] Lustre: fs26353-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12050.077755] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12060.633377] Lustre: server umount fs26353-OST0000 complete [12074.900339] Lustre: server umount fs26353-OST0001 complete [12095.327746] loop1: detected capacity change from 0 to 204800000 [12103.258479] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.586201] Lustre: fs26353-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12113.189608] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12121.911105] loop2: detected capacity change from 0 to 204800000 [12131.020808] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.509374] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12145.690338] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12145.702948] LustreError: Skipped 1 previous similar message [12155.722543] Lustre: server umount fs26353-OST0000 complete [12169.111142] Lustre: server umount fs26353-OST0001 complete [12171.237787] LustreError: 303733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.288697] LustreError: 303733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 76 previous similar messages [12186.594082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.611633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12186.619422] LustreError: Skipped 1 previous similar message [12186.623775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12186.631103] Lustre: Skipped 1 previous similar message [12191.716667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.833228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.856113] Lustre: Skipped 2 previous similar messages [12199.393159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12199.617044] Lustre: server umount lustre-OST0000 complete [12207.074366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12207.090669] Lustre: Skipped 1 previous similar message [12207.096900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12207.115070] Lustre: Skipped 1 previous similar message [12212.192693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12212.210691] Lustre: Skipped 1 previous similar message [12220.383138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12220.601636] Lustre: server umount lustre-MDT0000 complete [12224.273571] LustreError: 301885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475714 with bad export cookie 17117261322868078740 [12224.285193] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12224.726768] Lustre: server umount lustre-MDT0001 complete [12240.587094] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12244.473369] Key type lgssc unregistered [12244.695615] LNet: 315170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12244.699303] LNetError: 315170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12244.708461] LNet: Removed LNI 192.168.201.110@tcp [12245.551158] Key type .llcrypt unregistered [12245.552714] Key type ._llcrypt unregistered [12291.734569] Key type ._llcrypt registered [12291.736320] Key type .llcrypt registered [12291.852327] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12312.316480] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12314.200662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12314.239709] alg: No test for adler32 (adler32-zlib) [12316.516283] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12316.987075] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [12318.663174] Key type lgssc registered [12320.431414] Lustre: Echo OBD driver; http://www.lustre.org/ [12363.018134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12363.079837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.522852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12364.556856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12364.712211] Lustre: lustre-MDT0000: new disk, initializing [12364.845189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12364.867985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12369.287496] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12382.585139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.881477] Lustre: 319032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12383.030413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12383.039301] Lustre: Skipped 1 previous similar message [12383.208649] Lustre: lustre-MDT0001: new disk, initializing [12383.351289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12383.391507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12383.402044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12389.147657] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12400.669240] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12408.044991] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12415.140112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12415.344422] Lustre: lustre-OST0000: new disk, initializing [12415.346950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12415.350144] Lustre: 321003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12415.465492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12422.234695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12422.245337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12422.396785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12422.567776] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12432.432123] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12434.488455] Lustre: Mounted lustre-client - version 2.17.57_103_g02cdcd5 [12435.662930] Lustre: Unmounted lustre-client [12448.227632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12448.228219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12448.233237] LustreError: Skipped 1 previous similar message [12448.234892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.266507] Lustre: Skipped 1 previous similar message [12449.768511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12449.776251] Lustre: Skipped 1 previous similar message [12453.357491] Lustre: server umount lustre-OST0000 complete [12460.001853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12460.008168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12460.025201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12460.031190] Lustre: Skipped 1 previous similar message [12460.039964] Lustre: Skipped 2 previous similar messages [12464.607452] Lustre: server umount lustre-MDT0000 complete [12469.224810] LustreError: 319024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475959 with bad export cookie 6743002503031520335 [12469.229488] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12469.235756] LustreError: 319024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12469.515853] Lustre: server umount lustre-MDT0001 complete [12481.693231] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12484.805795] Key type lgssc unregistered [12486.239727] LNet: 323163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12486.254646] LNetError: 323163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12564.146435] LNet: Removed LNI 192.168.201.110@tcp [12565.720070] Key type .llcrypt unregistered [12565.722939] Key type ._llcrypt unregistered [12590.383667] Key type ._llcrypt registered [12590.388285] Key type .llcrypt registered [12590.574817] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [12606.476261] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12607.325936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12607.420315] alg: No test for adler32 (adler32-zlib) [12608.386126] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12608.570491] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [12610.256537] Key type lgssc registered [12611.465583] Lustre: Echo OBD driver; http://www.lustre.org/ [12650.650653] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:55:37 (1788476137) [12659.514809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12659.583956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12661.009074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12661.035755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12661.128300] Lustre: lustre-MDT0000: new disk, initializing [12661.212554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12661.239298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12666.738818] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12678.362920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.496536] Lustre: 326385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12678.647193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12678.658366] Lustre: Skipped 1 previous similar message [12678.778879] Lustre: lustre-MDT0001: new disk, initializing [12678.927651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12678.995430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12679.004290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12685.600922] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12698.976967] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12706.477298] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12714.476438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12714.841844] Lustre: lustre-OST0000: new disk, initializing [12714.844816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12714.861383] Lustre: 328354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12714.974707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12722.317488] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12724.381057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12724.403354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12724.510128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12733.778151] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12750.304094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12750.308109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12750.316020] LustreError: Skipped 1 previous similar message [12750.325216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.331891] Lustre: Skipped 1 previous similar message [12750.354042] Lustre: Skipped 1 previous similar message [12755.424838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12755.433095] Lustre: Skipped 1 previous similar message [12755.913486] Lustre: server umount lustre-OST0000 complete [12760.553326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12760.556142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.566106] Lustre: Skipped 1 previous similar message [12760.579159] Lustre: Skipped 1 previous similar message [12765.665252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.674235] Lustre: Skipped 1 previous similar message [12770.785868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12770.798313] Lustre: Skipped 1 previous similar message [12774.367233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12774.782802] Lustre: server umount lustre-MDT0000 complete [12778.813474] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476268 with bad export cookie 13575696797673006534 [12778.820431] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12778.822930] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12779.287082] Lustre: server umount lustre-MDT0001 complete [12790.940949] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [12793.365321] Key type lgssc unregistered [12793.649694] LNet: 330264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12793.654441] LNetError: 330264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12793.681860] LNet: Removed LNI 192.168.201.110@tcp [12794.586676] Key type .llcrypt unregistered [12794.589398] Key type ._llcrypt unregistered [12818.927229] Key type ._llcrypt registered [12818.933358] Key type .llcrypt registered [12819.061722] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [12821.103581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12821.348191] alg: No test for adler32 (adler32-zlib) [12822.769253] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [12823.093767] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [12824.863476] Key type lgssc registered [12826.910300] Lustre: Echo OBD driver; http://www.lustre.org/ [12836.262726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12836.294840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.873846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12843.156491] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12848.101228] LustreError: 331535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12848.120779] LustreError: 331535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12852.622338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12853.021472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12858.639801] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12868.560785] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12875.552744] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12882.626013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12883.012913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12889.611369] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12899.928532] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12912.044672] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12914.194643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12919.782769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12919.799093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.839602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.881758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.888809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12926.063807] Lustre: server umount lustre-MDT0000 complete [12927.972709] LustreError: 331535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.012483] LustreError: 331535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12930.745928] LustreError: 332271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476420 with bad export cookie 3363872975917933481 [12930.757344] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.758077] LustreError: 332271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12933.093397] LustreError: 331532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12933.095743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.101333] LustreError: 331532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12933.114283] Lustre: Skipped 1 previous similar message [12933.118203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12933.120986] Lustre: Skipped 3 previous similar messages [12935.138502] LustreError: 331532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12935.139814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12935.161522] LustreError: 331532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12935.188758] Lustre: Skipped 1 previous similar message [12940.257727] LustreError: 331529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12940.257792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12940.274795] LustreError: 331529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12940.299502] Lustre: Skipped 1 previous similar message [12945.375165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12945.682709] Lustre: server umount lustre-MDT0001 complete [12950.997739] Lustre: server umount lustre-OST0000 complete [12959.335276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12963.721053] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12966.102735] Lustre: 335719:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.110@tcp [12968.349873] Lustre: server umount MGS complete [12977.758816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.204953] LustreError: 336301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12978.230113] LustreError: 336301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12978.305584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12983.764502] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [12993.527920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.054452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12998.991557] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13008.835984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13009.195920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13016.349367] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13024.955081] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13026.715466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13033.287638] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13035.334163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13041.138517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13041.149748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.156986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13041.161550] Lustre: Skipped 3 previous similar messages [13042.921490] Lustre: server umount lustre-OST0000 complete [13049.313581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13049.343337] Lustre: Skipped 1 previous similar message [13053.391437] Lustre: server umount lustre-MDT0000 complete [13054.443403] LustreError: 336302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.468198] LustreError: 336302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13057.571975] LustreError: 337760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476547 with bad export cookie 3363872975917934321 [13057.584814] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13067.916969] Lustre: server umount lustre-MDT0001 complete [13080.145798] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13083.590995] Key type lgssc unregistered [13083.888047] LNet: 339737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13083.908076] LNetError: 339737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13084.978399] LNet: Removed LNI 192.168.201.110@tcp [13085.788410] Key type .llcrypt unregistered [13085.791245] Key type ._llcrypt unregistered [13106.108290] Key type ._llcrypt registered [13106.110053] Key type .llcrypt registered [13106.237408] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [13122.447960] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [13123.508523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13123.517888] alg: No test for adler32 (adler32-zlib) [13124.619718] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [13124.798325] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [13126.471205] Key type lgssc registered [13127.910622] Lustre: Echo OBD driver; http://www.lustre.org/ [13167.699334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13167.762159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.314945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13169.377807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13169.595306] Lustre: lustre-MDT0000: new disk, initializing [13169.722088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13169.768493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13175.291408] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13188.218356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.432032] Lustre: 342898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13188.561674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13188.575453] Lustre: Skipped 1 previous similar message [13188.685758] Lustre: lustre-MDT0001: new disk, initializing [13188.909463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13189.041315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13189.069023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13192.931693] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13204.775445] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13211.235224] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13218.673662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13219.007812] Lustre: lustre-OST0000: new disk, initializing [13219.011761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13219.017377] Lustre: 344866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13219.110802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13226.559292] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13228.599814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13228.608586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13228.727919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13236.907484] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13242.606319] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13242.844906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13248.042300] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13248.235780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13254.626775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13254.632147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13254.659047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13254.686397] Lustre: Skipped 1 previous similar message [13259.758668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13259.768019] Lustre: Skipped 1 previous similar message [13264.351402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13264.587842] Lustre: server umount lustre-OST0000 complete [13269.984934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13269.991157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13270.000022] Lustre: Skipped 1 previous similar message [13270.010502] Lustre: Skipped 2 previous similar messages [13275.110750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13275.115860] Lustre: Skipped 1 previous similar message [13280.232951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13280.246361] Lustre: Skipped 1 previous similar message [13283.295189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13283.580919] Lustre: server umount lustre-MDT0000 complete [13287.164087] LustreError: 343692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476776 with bad export cookie 16902781987643695502 [13287.166618] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13287.170278] LustreError: 343692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13287.540693] Lustre: server umount lustre-MDT0001 complete [13296.260761] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:06:23 (1788476783) [13313.619941] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [13322.804724] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [13361.024607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13361.467316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13361.516493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13361.634192] Lustre: lustre-MDT0000: new disk, initializing [13361.755118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13361.782304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13365.713339] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13377.834678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13378.013657] Lustre: 349629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13378.094440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13378.104093] Lustre: Skipped 1 previous similar message [13378.180924] Lustre: lustre-MDT0001: new disk, initializing [13378.271371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13378.322054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13378.344547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13383.008900] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13395.060488] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13404.290693] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13412.799634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13413.080803] Lustre: lustre-OST0000: new disk, initializing [13413.087228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13413.092695] Lustre: 351598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13413.179849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13420.182756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13420.202513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13420.321255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13420.358037] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [13429.459326] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13466.595569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13466.597295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13466.607448] LustreError: Skipped 2 previous similar messages [13466.609628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13466.619665] Lustre: Skipped 1 previous similar message [13466.637562] Lustre: Skipped 1 previous similar message [13472.387639] Lustre: server umount lustre-OST0000 complete [13476.833206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13476.842300] Lustre: Skipped 1 previous similar message [13482.852608] Lustre: server umount lustre-MDT0000 complete [13486.019666] LustreError: 349621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476975 with bad export cookie 16902781987643696384 [13486.022577] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13486.034350] LustreError: 349621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13486.378955] Lustre: server umount lustre-MDT0001 complete [13500.636098] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13503.240808] Key type lgssc unregistered [13503.622969] LNet: 353772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13503.627972] LNetError: 353772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13503.640570] LNet: Removed LNI 192.168.201.110@tcp [13504.671466] Key type .llcrypt unregistered [13504.675942] Key type ._llcrypt unregistered [13607.934600] Key type ._llcrypt registered [13607.936909] Key type .llcrypt registered [13608.032471] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13608.340434] Key type .llcrypt unregistered [13608.342222] Key type ._llcrypt unregistered [13696.285149] Key type ._llcrypt registered [13696.287960] Key type .llcrypt registered [13696.471269] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13696.759633] Key type .llcrypt unregistered [13696.761030] Key type ._llcrypt unregistered [13730.712036] Key type ._llcrypt registered [13730.713582] Key type .llcrypt registered [13730.892578] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13731.309444] Key type .llcrypt unregistered [13731.313942] Key type ._llcrypt unregistered [13765.410754] Key type ._llcrypt registered [13765.416924] Key type .llcrypt registered [13765.508959] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13765.955394] Key type .llcrypt unregistered [13765.960608] Key type ._llcrypt unregistered [13839.327155] Key type ._llcrypt registered [13839.330427] Key type .llcrypt registered [13839.532447] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13840.040670] Key type .llcrypt unregistered [13840.047548] Key type ._llcrypt unregistered [13873.810625] Key type ._llcrypt registered [13873.813372] Key type .llcrypt registered [13873.947971] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13874.303463] Key type .llcrypt unregistered [13874.306329] Key type ._llcrypt unregistered [13907.035291] Key type ._llcrypt registered [13907.038446] Key type .llcrypt registered [13907.191665] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13907.646424] Key type .llcrypt unregistered [13907.650322] Key type ._llcrypt unregistered [13941.483700] Key type ._llcrypt registered [13941.486560] Key type .llcrypt registered [13941.648687] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13941.945373] Key type .llcrypt unregistered [13941.947617] Key type ._llcrypt unregistered [13972.408448] Key type ._llcrypt registered [13972.410828] Key type .llcrypt registered [13972.543738] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [13972.936698] Key type .llcrypt unregistered [13972.938742] Key type ._llcrypt unregistered [14007.566478] Key type ._llcrypt registered [14007.568562] Key type .llcrypt registered [14007.714934] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [14008.061369] Key type .llcrypt unregistered [14008.062840] Key type ._llcrypt unregistered [14075.231497] Key type ._llcrypt registered [14075.234412] Key type .llcrypt registered [14075.396389] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [14075.758504] Key type .llcrypt unregistered [14075.764212] Key type ._llcrypt unregistered [14110.452030] Key type ._llcrypt registered [14110.456566] Key type .llcrypt registered [14110.668371] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [14111.045571] Key type .llcrypt unregistered [14111.048409] Key type ._llcrypt unregistered [14145.102569] Key type ._llcrypt registered [14145.104336] Key type .llcrypt registered [14145.292282] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [14145.822064] Key type .llcrypt unregistered [14145.825307] Key type ._llcrypt unregistered [14205.646487] Key type ._llcrypt registered [14205.650284] Key type .llcrypt registered [14205.809853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [14222.190027] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [14223.842421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14223.956257] alg: No test for adler32 (adler32-zlib) [14224.888062] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [14225.089889] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [14226.759208] Key type lgssc registered [14228.130947] Lustre: Echo OBD driver; http://www.lustre.org/ [14267.351399] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14267.376784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14268.730321] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14268.771360] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14268.914309] Lustre: tst250fs-MDT0000: new disk, initializing [14269.024335] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14269.049359] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14274.187669] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [14285.115472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14285.268413] Lustre: 374059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14285.350462] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14285.354517] Lustre: Skipped 1 previous similar message [14285.446612] Lustre: tst250fs-MDT0001: new disk, initializing [14285.556036] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14285.619187] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14285.639233] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14292.146790] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [14297.402570] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14304.411691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14304.776960] Lustre: tst250fs-OST0000: new disk, initializing [14304.783439] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14304.789849] Lustre: 375692:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14304.902584] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14305.904267] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14305.918394] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14306.106099] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14311.466152] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [14323.789915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14323.971429] Lustre: tst250fs-OST0001: new disk, initializing [14323.978970] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14323.986054] Lustre: 376563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14324.094877] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14330.980474] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14330.998184] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14331.158646] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14331.236443] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [14336.914718] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14338.967837] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14338.975877] Lustre: Skipped 1 previous similar message [14362.090147] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14362.096799] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14362.109840] Lustre: Skipped 1 previous similar message [14362.125558] Lustre: Skipped 3 previous similar messages [14367.217990] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14367.226016] Lustre: Skipped 3 previous similar messages [14372.326273] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14372.336076] Lustre: Skipped 3 previous similar messages [14373.343809] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14373.518220] Lustre: server umount tst250fs-MDT0000 complete [14377.895821] LustreError: 374051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477867 with bad export cookie 3794709685040254395 [14377.898105] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14377.904248] LustreError: 374051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14378.267465] Lustre: server umount tst250fs-MDT0001 complete [14393.281986] Lustre: server umount tst250fs-OST0000 complete [14407.972147] Lustre: server umount tst250fs-OST0001 complete [14429.238801] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [14437.394609] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [14472.785569] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:26:00 (1788477960) [14474.723121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14492.890984] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [14502.968590] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [14537.488622] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14227 sec ========= 19:27:04 (1788478024) [14539.431499] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:27:06 (1788478026) === [14543.473331] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:27:10 (1788478030) === [14579.378837] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [14581.814530] Key type lgssc unregistered [14582.112856] LNet: 385694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14582.120872] LNetError: 385694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14582.136156] LNet: Removed LNI 192.168.201.110@tcp [14582.913275] Key type .llcrypt unregistered [14582.916486] Key type ._llcrypt unregistered