[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502737775 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007636] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.021702] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026417] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030140] *** VALIDATE tmpfs *** [ 0.031448] *** VALIDATE proc *** [ 0.032263] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042039] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.044940] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053290] rcu: Hierarchical SRCU implementation. [ 0.055488] smp: Bringing up secondary CPUs ... [ 0.056590] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060693] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143053] node 0 deferred pages initialised in 78ms [ 0.145116] devtmpfs: initialized [ 0.146221] x86/mm: Memory block size: 128MB [ 0.148216] gcov: version magic: 0x41383552 [ 0.150063] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151057] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152289] pinctrl core: initialized pinctrl subsystem [ 0.153142] [ 0.154015] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156012] ** ** [ 0.157013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158017] ** ** [ 0.159012] ** This means that this kernel is built to expose internal ** [ 0.160011] ** IOMMU data structures, which may compromise security on ** [ 0.161014] ** your system. ** [ 0.162014] ** ** [ 0.163015] ** If you see this message and you are not debugging the ** [ 0.164012] ** kernel, report this immediately to your vendor! ** [ 0.165012] ** ** [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167012] ************************************************************* [ 0.168628] NET: Registered protocol family 16 [ 0.169287] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172410] cpuidle: using governor menu [ 0.174676] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177487] PCI: Using configuration type 1 for base access [ 0.179130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191189] cryptd: max_cpu_qlen set to 1000 [ 0.193063] ACPI: Added _OSI(Module Device) [ 0.194015] ACPI: Added _OSI(Processor Device) [ 0.195012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200027] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202397] ACPI: Interpreter enabled [ 0.203045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203906] ACPI: Using IOAPIC for interrupt routing [ 0.204121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205305] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213802] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218084] acpiphp: Slot [2] registered [ 0.219141] acpiphp: Slot [5] registered [ 0.220106] acpiphp: Slot [6] registered [ 0.220879] acpiphp: Slot [7] registered [ 0.221068] acpiphp: Slot [8] registered [ 0.221843] acpiphp: Slot [9] registered [ 0.222519] acpiphp: Slot [10] registered [ 0.223076] acpiphp: Slot [3] registered [ 0.224064] acpiphp: Slot [4] registered [ 0.225065] acpiphp: Slot [11] registered [ 0.227071] acpiphp: Slot [12] registered [ 0.228089] acpiphp: Slot [13] registered [ 0.229071] acpiphp: Slot [14] registered [ 0.230097] acpiphp: Slot [15] registered [ 0.232189] acpiphp: Slot [16] registered [ 0.233107] acpiphp: Slot [17] registered [ 0.235101] acpiphp: Slot [18] registered [ 0.236135] acpiphp: Slot [19] registered [ 0.238371] acpiphp: Slot [20] registered [ 0.239080] acpiphp: Slot [21] registered [ 0.241093] acpiphp: Slot [22] registered [ 0.242089] acpiphp: Slot [23] registered [ 0.243183] acpiphp: Slot [24] registered [ 0.245092] acpiphp: Slot [25] registered [ 0.246084] acpiphp: Slot [26] registered [ 0.247070] acpiphp: Slot [27] registered [ 0.248076] acpiphp: Slot [28] registered [ 0.249064] acpiphp: Slot [29] registered [ 0.250009] acpiphp: Slot [30] registered [ 0.250993] acpiphp: Slot [31] registered [ 0.252079] PCI host bridge to bus 0000:00 [ 0.252993] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.255023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.257019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.259066] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.260015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.262017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.263177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268076] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276605] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.281553] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.283014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.285017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.289654] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.292778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.294035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.296688] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.313017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.317871] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.322335] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.329017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.364473] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.369015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.375017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.387020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.396156] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.409029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.430022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.457018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.470577] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.477016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.483298] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.500021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.509062] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.522016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.536015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.556021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.572495] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.584016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.589018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.608017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.623788] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.626336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628339] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.635234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.639115] iommu: Default domain type: Passthrough [ 0.641111] SCSI subsystem initialized [ 0.642149] ACPI: bus type USB registered [ 0.644087] usbcore: registered new interface driver usbfs [ 0.645113] usbcore: registered new interface driver hub [ 0.647205] usbcore: registered new device driver usb [ 0.649266] pps_core: LinuxPPS API ver. 1 registered [ 0.651012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.653069] PTP clock support registered [ 0.655083] EDAC MC: Ver: 3.0.0 [ 0.656644] PCI: Using ACPI for IRQ routing [ 0.658866] NetLabel: Initializing [ 0.660010] NetLabel: domain hash size = 128 [ 0.662020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.664068] NetLabel: unlabeled traffic allowed by default [ 0.666094] vgaarb: loaded [ 0.666587] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.667008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.673090] clocksource: Switched to clocksource kvm-clock [ 0.782871] VFS: Disk quotas dquot_6.6.0 [ 0.784606] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.787393] *** VALIDATE ramfs *** [ 0.788840] *** VALIDATE hugetlbfs *** [ 0.791224] pnp: PnP ACPI init [ 0.794481] pnp: PnP ACPI: found 6 devices [ 0.820795] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.823537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.825444] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.827209] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.829373] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.831386] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833170] NET: Registered protocol family 2 [ 0.835119] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.839054] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.841366] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.847071] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.850780] TCP: Hash tables configured (established 65536 bind 65536) [ 0.853797] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.857118] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860121] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.863348] NET: Registered protocol family 1 [ 0.866114] RPC: Registered named UNIX socket transport module. [ 0.868710] RPC: Registered udp transport module. [ 0.871071] RPC: Registered tcp transport module. [ 0.873163] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.876042] NET: Registered protocol family 44 [ 0.878036] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.880310] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.882758] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.885351] PCI: CLS 0 bytes, default 64 [ 0.887299] Unpacking initramfs... [ 2.255841] debug: unmapping init [mem 0xffff8f3d3cc54000-0xffff8f3d3ffbffff] [ 2.258745] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.260406] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.262335] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.774519] Initialise system trusted keyrings [ 2.775985] Key type blacklist registered [ 2.777915] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.789430] zbud: loaded [ 2.792978] *** VALIDATE nfs *** [ 2.794466] *** VALIDATE nfs4 *** [ 2.796873] pstore: using deflate compression [ 2.801407] Platform Keyring initialized [ 2.922803] NET: Registered protocol family 38 [ 2.924860] Key type asymmetric registered [ 2.926315] Asymmetric key parser 'x509' registered [ 2.927811] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.930980] io scheduler mq-deadline registered [ 2.932908] io scheduler kyber registered [ 2.934902] io scheduler bfq registered [ 2.936975] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.940372] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.943496] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.946847] ACPI: Power Button [PWRF] [ 2.952541] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.960263] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.972786] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.979126] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.991183] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.024525] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.053768] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.059494] Non-volatile memory driver v1.3 [ 3.061407] Linux agpgart interface v0.103 [ 3.099872] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.102710] vda: detected capacity change from 0 to 74706944 [ 3.122856] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.125614] vdb: detected capacity change from 0 to 1073741824 [ 3.146477] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.149266] vdc: detected capacity change from 0 to 2621440000 [ 3.174854] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177639] vdd: detected capacity change from 0 to 2621440000 [ 3.192993] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.196300] vde: detected capacity change from 0 to 4294967296 [ 3.218389] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.220792] vdf: detected capacity change from 0 to 4294967296 [ 3.240070] libphy: Fixed MDIO Bus: probed [ 3.245391] usbcore: registered new interface driver usbserial_generic [ 3.247971] usbserial: USB Serial support registered for generic [ 3.250699] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.255350] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.257270] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.259995] mousedev: PS/2 mouse device common for all mice [ 3.263925] rtc_cmos 00:05: RTC can wake from S4 [ 3.264153] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.266778] rtc_cmos 00:05: registered as rtc0 [ 3.271214] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.272701] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.274715] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.279923] intel_pstate: CPU model not supported [ 3.283516] hid: raw HID events driver (C) Jiri Kosina [ 3.285817] usbcore: registered new interface driver usbhid [ 3.287704] usbhid: USB HID core driver [ 3.289307] drop_monitor: Initializing network drop monitor service [ 3.291555] Initializing XFRM netlink socket [ 3.293582] NET: Registered protocol family 10 [ 3.296766] Segment Routing with IPv6 [ 3.298520] NET: Registered protocol family 17 [ 3.301515] mpls_gso: MPLS GSO support [ 3.305709] RAS: Correctable Errors collector initialized. [ 3.307482] AVX version of gcm_enc/dec engaged. [ 3.309104] AES CTR mode by8 optimization enabled [ 3.379916] sched_clock: Marking stable (3379809830, 0)->(4249179838, -869370008) [ 3.386946] registered taskstats version 1 [ 3.389875] Loading compiled-in X.509 certificates [ 3.391733] zswap: loaded using pool lzo/zbud [ 3.420814] Key type big_key registered [ 3.435244] Key type encrypted registered [ 3.436903] ima: No TPM chip found, activating TPM-bypass! [ 3.439151] ima: Allocated hash algorithm: sha1 [ 3.441089] ima: No architecture policies found [ 3.443142] evm: Initialising EVM extended attributes: [ 3.445100] evm: security.selinux [ 3.446637] evm: security.ima [ 3.447812] evm: security.capability [ 3.449211] evm: HMAC attrs: 0x1 [ 3.452208] rtc_cmos 00:05: setting system clock to 2026-08-13 19:10:23 UTC (1786648223) [ 3.459646] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.462706] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.471088] Write protecting the kernel read-only data: 28672k [ 3.474634] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.477085] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.511700] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.519227] systemd[1]: Detected virtualization kvm. [ 3.520732] systemd[1]: Detected architecture x86-64. [ 3.522866] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.548704] systemd[1]: No hostname configured. [ 3.550318] systemd[1]: Set hostname to . [ 3.552166] random: systemd: uninitialized urandom read (16 bytes read) [ 3.553783] systemd[1]: Initializing machine ID from random generator. [ 3.705074] random: systemd: uninitialized urandom read (16 bytes read) [ 3.708483] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.718442] random: systemd: uninitialized urandom read (16 bytes read) [ 3.721751] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.726508] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.411541] device-mapper: uevent: version 1.0.3 [ 4.414712] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.220355] virtio_net virtio0 ens2: renamed from eth0 [ 5.281119] random: fast init done [ 5.408091] scsi host0: ata_piix [ 5.439164] scsi host1: ata_piix [ 5.440742] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.443212] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.312694] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.955693] random: crng init done [ 9.957025] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.585779] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.876805] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.171954] SELinux: Disabled at runtime. [ 12.231416] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.239858] systemd[1]: Detected virtualization kvm. [ 12.241867] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.781201] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.784409] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.789414] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.793239] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.796564] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.804880] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.810527] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ 12.921628] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.323995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.715363] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.750176] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.981177] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.990572] EDAC sbridge: Ver: 1.1.2 [ 15.734971] Key type dns_resolver registered [ 16.049173] NFS: Registering the id_resolver key type [ 16.050506] Key type id_resolver registered [ 16.051418] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg139-server login: [ 41.712722] libcfs: loading out-of-tree module taints kernel. [ 41.737095] Key type ._llcrypt registered [ 41.738740] Key type .llcrypt registered [ 41.785656] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 56.562956] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 58.359489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.394985] alg: No test for adler32 (adler32-zlib) [ 60.202522] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 61.617113] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 63.416182] Key type lgssc registered [ 65.288838] hrtimer: interrupt took 3708879 ns [ 66.509616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.256080] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.599277] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 150.995676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.052511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.355472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.377462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.520321] Lustre: lustre-MDT0000: new disk, initializing [ 152.670093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.684760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.456414] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.707310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.839688] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.891320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.896580] Lustre: Skipped 1 previous similar message [ 170.965716] Lustre: lustre-MDT0001: new disk, initializing [ 171.119220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.159426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.182547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.435994] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.337424] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.521983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.236742] Lustre: lustre-OST0000: new disk, initializing [ 196.261981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.278822] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.465971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.307188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.324810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.448720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.847609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.071311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.241827] Lustre: lustre-OST0001: new disk, initializing [ 220.256337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.273736] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.403909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.897835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.924741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.084880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 227.962535] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.632716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.180426] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.636752] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.288995] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing yml_node [ 271.322774] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 275.419894] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 278.159891] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 279.456677] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 15:14:58 EDT 2026 [ 299.409545] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 301.149405] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.432914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.437823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.461282] Lustre: Skipped 3 previous similar messages [ 318.486559] Lustre: Skipped 3 previous similar messages [ 323.556201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.562723] Lustre: Skipped 3 previous similar messages [ 324.769397] Lustre: server umount lustre-MDT0000 complete [ 328.302286] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648548 with bad export cookie 8777219670900110620 [ 328.305450] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.309468] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.673658] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.690734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.709834] Lustre: Skipped 1 previous similar message [ 333.797489] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.805874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.832787] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 333.864532] Lustre: Skipped 1 previous similar message [ 334.746776] Lustre: server umount lustre-MDT0001 complete [ 338.984833] Lustre: server umount lustre-OST0000 complete [ 342.690130] Lustre: server umount lustre-OST0001 complete [ 350.209437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 359.366975] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 395.010307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.262771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.285735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.382115] Lustre: lustre-MDT0000: new disk, initializing [ 395.454296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.465785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.739933] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 412.322669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.487512] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.542805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.545932] Lustre: Skipped 1 previous similar message [ 412.607550] Lustre: lustre-MDT0001: new disk, initializing [ 412.726713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.765612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.786400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.466820] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 428.457282] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.545660] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.213922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.815347] Lustre: lustre-OST0000: new disk, initializing [ 442.832857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.843847] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.113555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.904603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.922350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.159084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.753494] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 470.695682] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 477.312794] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.590271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.777058] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 483.115553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.345410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.355047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.360082] LustreError: Skipped 1 previous similar message [ 485.363323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.393972] Lustre: Skipped 2 previous similar messages [ 492.018439] Lustre: server umount lustre-OST0000 complete [ 498.667714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.676390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.684641] Lustre: Skipped 1 previous similar message [ 498.702771] Lustre: Skipped 4 previous similar messages [ 503.605519] Lustre: server umount lustre-MDT0000 complete [ 503.787272] LustreError: 15090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.813236] LustreError: 15090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 508.210986] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648728 with bad export cookie 8777219670900112160 [ 508.232410] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 508.241250] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.078267] Lustre: server umount lustre-MDT0001 complete [ 518.612834] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:18:56 (1786648736) [ 543.846268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.400497] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.479781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.447594] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 549.865810] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.965234] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.767417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.767469] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 577.563543] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.640301] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.745306] Lustre: Failing over lustre-MDT0000 [ 588.046163] Lustre: server umount lustre-MDT0000 complete [ 588.775417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.821928] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.852907] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 604.002725] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786648808/real 1786648808] req@ffff8f3c83b24e00 x1873436508069504/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786648824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 604.039207] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.166203] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.185493] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 608.711883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.304992] LustreError: 3643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f3dc1b5f100 x1873436508072064/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.710905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.722704] Lustre: Skipped 1 previous similar message [ 614.801856] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.020478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.044776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.117747] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.976724] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 632.265990] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 638.437760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.440896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.461857] Lustre: Skipped 2 previous similar messages [ 638.479993] Lustre: Skipped 1 previous similar message [ 643.564900] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.573857] LustreError: 19925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 643.715376] Lustre: server umount lustre-MDT0000 complete [ 647.989835] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648868 with bad export cookie 8777219670900113574 [ 648.001544] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.008550] LustreError: 20660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 648.703584] Lustre: server umount lustre-MDT0001 complete [ 658.288852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.962230] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 675.232804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.492597] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 689.547175] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.861727] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.813149] Lustre: Failing over lustre-MDT0000 [ 696.823274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.844041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.098252] Lustre: server umount lustre-MDT0000 complete [ 711.138810] LustreError: 23996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.185426] LustreError: 23996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 717.288503] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786648921/real 1786648921] req@ffff8f3dafa58a80 x1873436508100608/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786648937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 717.331543] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.014412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.557755] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x79cef2c977928450 [ 728.570735] Lustre: MGC192.168.201.139@tcp: Connection restored to 0@lo (at 0@lo) [ 728.585963] Lustre: Skipped 1 previous similar message [ 728.977951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 734.180421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 734.183752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 734.246176] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 735.111525] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 748.835438] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 758.766297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.789475] Lustre: Skipped 1 previous similar message [ 763.681465] Lustre: server umount lustre-MDT0000 complete [ 767.750120] LustreError: 23250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648987 with bad export cookie 8777219670900114512 [ 767.764029] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.038793] Lustre: server umount lustre-MDT0001 complete [ 776.918480] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:23:16 (1786648996) [ 796.039588] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 808.623883] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 863.327463] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 875.134557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.308942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 875.358369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 875.477414] Lustre: lustre-MDT0000: new disk, initializing [ 875.564947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.567907] Lustre: Skipped 3 previous similar messages [ 875.581516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 879.597442] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 889.970427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.130368] Lustre: 29662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 890.455931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 896.326032] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 906.658768] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 912.490592] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 919.642916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 919.975305] Lustre: 31635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 921.243229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 921.359992] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 921.490448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 921.527440] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 926.628901] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 936.920768] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.213501] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 967.654191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.657894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.661102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.661111] Lustre: Skipped 2 previous similar messages [ 967.667770] LustreError: Skipped 1 previous similar message [ 967.696613] Lustre: Skipped 2 previous similar messages [ 977.377729] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 977.874121] Lustre: server umount lustre-OST0000 complete [ 982.511353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.934563] Lustre: server umount lustre-MDT0000 complete [ 992.555859] LustreError: 29657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649212 with bad export cookie 8777219670900114911 [ 992.569957] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.579668] LustreError: 29657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 993.037763] Lustre: server umount lustre-MDT0001 complete [ 1006.432578] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1009.622709] Key type lgssc unregistered [ 1010.147751] LNet: 33617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.165563] LNetError: 33617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.190681] LNet: Removed LNI 192.168.201.139@tcp [ 1011.458897] Key type .llcrypt unregistered [ 1011.466267] Key type ._llcrypt unregistered [ 1037.674662] Key type ._llcrypt registered [ 1037.676865] Key type .llcrypt registered [ 1037.847814] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1038.897614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.077738] alg: No test for adler32 (adler32-zlib) [ 1040.213920] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1040.441108] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1042.104181] Key type lgssc registered [ 1043.252643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.086250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1051.124882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.704402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.195674] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1062.897746] LustreError: 35034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.922073] LustreError: 35034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1065.343175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.675640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.493572] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1078.197698] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.209146] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1091.386964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.838391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.011022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1098.357637] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1106.351478] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1110.272906] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1123.727543] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:29:02 (1786649342) [ 1125.726693] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1127.755730] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1134.578558] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:29:13 (1786649353) [ 1144.358886] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:29:23 (1786649363) [ 1145.424764] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1154.754219] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1154.762895] Lustre: Skipped 2 previous similar messages [ 1156.603228] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:29:35 (1786649375) [ 1160.525061] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1160.529100] Lustre: Skipped 3 previous similar messages [ 1178.871570] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1178.874907] Lustre: Skipped 6 previous similar messages [ 1181.628546] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:30:00 (1786649400) [ 1184.361552] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1188.141628] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1197.578422] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:30:16 (1786649416) [ 1199.891391] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1199.898278] Lustre: Skipped 3 previous similar messages [ 1215.072750] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:30:33 (1786649433) [ 1233.379657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1233.385055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.396028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.408674] Lustre: Skipped 1 previous similar message [ 1238.505883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.512992] Lustre: Skipped 2 previous similar messages [ 1238.709812] Lustre: server umount lustre-OST0000 complete [ 1243.618296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.622484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.636223] Lustre: Skipped 1 previous similar message [ 1248.740070] LustreError: 39884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.769367] LustreError: 39884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1248.899479] Lustre: server umount lustre-MDT0000 complete [ 1253.364448] LustreError: 35767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649473 with bad export cookie 729065464273453120 [ 1253.372255] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.390337] LustreError: 35767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1253.902862] Lustre: server umount lustre-MDT0001 complete [ 1265.940228] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1268.739347] Key type lgssc unregistered [ 1269.125044] LNet: 42895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.141230] LNetError: 42895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.165331] LNet: Removed LNI 192.168.201.139@tcp [ 1270.092632] Key type .llcrypt unregistered [ 1270.095202] Key type ._llcrypt unregistered [ 1289.299294] Key type ._llcrypt registered [ 1289.313856] Key type .llcrypt registered [ 1289.533914] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 1305.948468] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1307.246742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.271102] alg: No test for adler32 (adler32-zlib) [ 1308.413218] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1308.771377] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1310.472171] Key type lgssc registered [ 1311.758826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.735810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1344.758282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.927705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1345.968363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1346.069973] Lustre: lustre-MDT0000: new disk, initializing [ 1346.143608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.154839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1350.029853] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1359.286594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.464819] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1359.548374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1359.555126] Lustre: Skipped 1 previous similar message [ 1359.643839] Lustre: lustre-MDT0001: new disk, initializing [ 1359.722465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.764109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1359.779792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1364.520633] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1374.218929] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.920882] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.074472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1387.290607] Lustre: lustre-OST0000: new disk, initializing [ 1387.296089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1387.304049] Lustre: 48014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1387.411742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.212746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1393.234051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1393.409783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1394.352972] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1404.020314] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.193156] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1409.407227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1414.040544] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1414.293940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1419.233133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.246328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.251512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.270141] Lustre: Skipped 1 previous similar message [ 1422.481414] Lustre: server umount lustre-OST0000 complete [ 1426.414864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1426.425931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.442671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.447317] Lustre: Skipped 1 previous similar message [ 1429.478508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.482215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.685846] Lustre: server umount lustre-MDT0000 complete [ 1436.193751] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649656 with bad export cookie 11372274426468428489 [ 1436.194438] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.208153] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1436.538601] Lustre: server umount lustre-MDT0001 complete [ 1446.106706] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:34:24 (1786649664) [ 1452.717117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.111713] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.127446] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1453.175209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.472479] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1458.674558] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.791757] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.455540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.697552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1472.292441] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1480.279558] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1486.026272] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1494.572544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.114455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.120180] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1510.750611] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1518.580697] LustreError: 53009:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1519.637150] Lustre: Modifying parameter general.timeout=1 in log params [ 1521.674822] Lustre: Modifying parameter general.timeout=3 in log params [ 1521.678710] Lustre: Skipped 1 previous similar message [ 1524.721098] Lustre: Modifying parameter general.timeout=6 in log params [ 1524.724802] Lustre: Skipped 2 previous similar messages [ 1529.462664] Lustre: Modifying parameter general.timeout=10 in log params [ 1529.467194] Lustre: Skipped 3 previous similar messages [ 1537.759833] Lustre: Modifying parameter general.timeout=18 in log params [ 1537.768312] Lustre: Skipped 7 previous similar messages [ 1553.797281] Lustre: Modifying parameter general.timeout=34 in log params [ 1553.811466] Lustre: Skipped 15 previous similar messages [ 1586.195193] Lustre: Modifying parameter general.timeout=68 in log params [ 1586.200472] Lustre: Skipped 33 previous similar messages [ 1649.399365] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:37:48 (1786649868) [ 1650.607282] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1650.613241] Lustre: Skipped 62 previous similar messages [ 1652.475902] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1654.048909] Lustre: 60053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1657.601904] Lustre: 60250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1657.610588] Lustre: 60250:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1687.304335] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1687.316929] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1687.350971] LustreError: 61822:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1691.197094] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:38:30 (1786649910) [ 1703.395441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.404075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.412591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.416152] Lustre: Skipped 1 previous similar message [ 1709.514281] Lustre: server umount lustre-MDT0000 complete [ 1712.098848] LustreError: 51491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.136238] LustreError: 51491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1712.821621] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649932 with bad export cookie 11372274426468429371 [ 1712.829676] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1712.831484] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1713.147958] Lustre: server umount lustre-MDT0001 complete [ 1726.249222] Lustre: server umount lustre-OST0000 complete [ 1752.498693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.767958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1753.090466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.511924] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1765.969930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.061466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1766.079293] Lustre: 64354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.717974] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1780.281812] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1786.084886] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1792.475527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1792.749747] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1793.004736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.008973] Lustre: Skipped 1 previous similar message [ 1799.331580] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1807.351585] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1809.337186] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1809.346437] Lustre: Skipped 22 previous similar messages [ 1809.400162] Lustre: 66921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1821.159100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1821.166158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.175257] Lustre: Skipped 2 previous similar messages [ 1821.189901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.197473] Lustre: Skipped 3 previous similar messages [ 1824.578727] Lustre: server umount lustre-OST0000 complete [ 1831.917204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.927840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.941267] Lustre: Skipped 2 previous similar messages [ 1831.950373] Lustre: Skipped 2 previous similar messages [ 1834.733771] Lustre: server umount lustre-MDT0000 complete [ 1837.026109] LustreError: 64364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.046293] LustreError: 64364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1838.447313] LustreError: 66216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650058 with bad export cookie 11372274426468432416 [ 1838.449994] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.454160] LustreError: 66216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1838.782838] Lustre: server umount lustre-MDT0001 complete [ 1848.293496] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1850.749236] Key type lgssc unregistered [ 1851.040146] LNet: 68136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.045912] LNetError: 68136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.072446] LNet: Removed LNI 192.168.201.139@tcp [ 1851.964597] Key type .llcrypt unregistered [ 1851.967048] Key type ._llcrypt unregistered [ 1876.137405] Key type ._llcrypt registered [ 1876.139736] Key type .llcrypt registered [ 1876.250797] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1877.532787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1877.663581] alg: No test for adler32 (adler32-zlib) [ 1878.817587] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1879.109394] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1880.808190] Key type lgssc registered [ 1882.068243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1890.188641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1890.236390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.723726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.084076] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1901.035537] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.068674] LustreError: 69551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1904.530154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.994914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.519932] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1916.628287] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1922.996736] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.103162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.727151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.355335] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1944.162252] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.430098] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1964.518892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.527410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.540858] Lustre: Skipped 1 previous similar message [ 1964.560992] Lustre: Skipped 2 previous similar messages [ 1969.636792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.645951] Lustre: Skipped 2 previous similar messages [ 1969.814527] Lustre: server umount lustre-MDT0000 complete [ 1973.783935] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650193 with bad export cookie 4234088093778468430 [ 1973.792623] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1973.797718] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.139086] Lustre: server umount lustre-MDT0001 complete [ 1988.419755] Lustre: server umount lustre-OST0000 complete [ 2019.241237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.703168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2019.755079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2020.154126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.632270] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2034.604537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.763180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2034.813814] Lustre: 74421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2035.178481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2039.574903] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2047.377480] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2052.178250] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.421436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.698138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2059.116087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.377739] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2071.881483] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.561891] Lustre: 76993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2075.434890] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2083.299963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.306648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.308653] Lustre: Skipped 2 previous similar messages [ 2087.335742] Lustre: server umount lustre-OST0000 complete [ 2091.495895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2091.501364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.517518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.524793] Lustre: Skipped 1 previous similar message [ 2093.540198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.658656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.665949] Lustre: Skipped 3 previous similar messages [ 2105.318382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2105.478959] Lustre: server umount lustre-MDT0000 complete [ 2108.902532] LustreError: 74434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.919453] LustreError: 74434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2109.561499] LustreError: 74415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650329 with bad export cookie 4234088093778469277 [ 2109.563615] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.572313] LustreError: 74415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2109.904387] Lustre: server umount lustre-MDT0001 complete [ 2123.829337] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2126.564960] Key type lgssc unregistered [ 2126.881700] LNet: 78211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.895192] LNetError: 78211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.912566] LNet: Removed LNI 192.168.201.139@tcp [ 2127.849261] Key type .llcrypt unregistered [ 2127.851507] Key type ._llcrypt unregistered [ 2154.281615] Key type ._llcrypt registered [ 2154.289860] Key type .llcrypt registered [ 2154.428554] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2155.419449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.475547] alg: No test for adler32 (adler32-zlib) [ 2156.633172] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2156.842867] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2158.496293] Key type lgssc registered [ 2159.431744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2166.198173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2166.213397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.608228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.145976] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2177.003934] LustreError: 79624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.012741] LustreError: 79624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2180.463478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.046503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2185.773253] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2193.090521] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.641609] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.095981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.551188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.344880] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2221.245237] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2228.132791] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2229.054093] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2230.940646] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2230.947178] Lustre: Skipped 1 previous similar message [ 2233.374958] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2233.382474] Lustre: Skipped 2 previous similar messages [ 2237.942433] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2237.948614] Lustre: Skipped 4 previous similar messages [ 2246.854767] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2246.857796] Lustre: Skipped 8 previous similar messages [ 2263.313413] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.316896] Lustre: Skipped 16 previous similar messages [ 2295.359557] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2295.362436] Lustre: Skipped 30 previous similar messages [ 2330.775730] Lustre: 87723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2333.535652] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:49:12 (1786650552) [ 2335.300846] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2337.222266] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:49:16 (1786650556) [ 2344.937297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.941516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.958753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2345.957347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.960330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.967800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2347.442959] Lustre: server umount lustre-OST0000 complete [ 2354.659099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2354.670091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.689743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.704955] Lustre: Skipped 1 previous similar message [ 2357.626589] Lustre: server umount lustre-MDT0000 complete [ 2361.313843] LustreError: 79621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.336900] LustreError: 79621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2361.797784] LustreError: 79604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650581 with bad export cookie 1513010554172772768 [ 2361.813203] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.227922] Lustre: server umount lustre-MDT0001 complete [ 2373.731828] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2376.631939] Key type lgssc unregistered [ 2376.928784] LNet: 88975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.934251] LNetError: 88975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.960943] LNet: Removed LNI 192.168.201.139@tcp [ 2378.983924] Key type .llcrypt unregistered [ 2378.986206] Key type ._llcrypt unregistered [ 2384.498423] Key type ._llcrypt registered [ 2384.505339] Key type .llcrypt registered [ 2384.749042] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2392.137614] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules [ 2394.304257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.367390] alg: No test for adler32 (adler32-zlib) [ 2395.894291] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2396.527390] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2398.473987] Key type lgssc registered [ 2400.287277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.721201] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2419.709624] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2424.129416] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2430.186844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.209483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.816494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.270170] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2441.190195] LustreError: 91252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.216196] LustreError: 91252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2446.090581] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:51:05 (1786650665) [ 2446.310208] LustreError: 91251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.171129] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2450.188757] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2450.197173] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2450.651923] Lustre: server umount lustre-MDT0000 complete [ 2465.268325] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2467.868972] Key type lgssc unregistered [ 2468.156151] LNet: 92781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.165562] LNetError: 92781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.197062] LNet: Removed LNI 192.168.201.139@tcp [ 2469.136886] Key type .llcrypt unregistered [ 2469.141978] Key type ._llcrypt unregistered [ 2487.711598] Key type ._llcrypt registered [ 2487.717601] Key type .llcrypt registered [ 2487.797171] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2489.536850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2489.648274] alg: No test for adler32 (adler32-zlib) [ 2490.688641] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2490.877974] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2492.528246] Key type lgssc registered [ 2493.865808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.812913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.836220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.199195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.554204] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2512.357457] LustreError: 94005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.368545] LustreError: 94005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2514.499708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.169460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2519.846747] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2527.607555] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.076366] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2540.729666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2541.384207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.219242] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2557.277716] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2605.013708] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:53:44 (1786650824) [ 2606.662344] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2608.184152] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:53:47 (1786650827) [ 2616.290456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.298874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.304645] Lustre: Skipped 1 previous similar message [ 2616.316481] Lustre: Skipped 2 previous similar messages [ 2618.703170] Lustre: server umount lustre-MDT0000 complete [ 2621.419076] LustreError: 94736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.459449] LustreError: 94736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2622.252314] LustreError: 93987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650842 with bad export cookie 757835464931604193 [ 2622.260254] LustreError: 93987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2622.260933] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.607786] Lustre: server umount lustre-MDT0001 complete [ 2636.798194] Lustre: server umount lustre-OST0000 complete [ 2650.489948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.894109] LustreError: 98281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.079040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.458972] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2656.226435] LustreError: 98282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.246198] LustreError: 98282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2660.328471] LustreError: 98281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.928434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.378464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2669.487245] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2679.205820] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.207711] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2701.810539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.036031] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2702.043717] LustreError: 98270:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2702.068576] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2702.076481] LustreError: 100365:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2702.086863] LustreError: 100365:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2702.095069] LustreError: 100365:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2702.170857] Lustre: server umount lustre-OST0000 complete [ 2702.174567] LustreError: 100365:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2711.748777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.906474] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2711.919420] LustreError: 98268:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2711.937801] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2711.950812] LustreError: 100830:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2711.958443] LustreError: 100830:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2711.962404] LustreError: 100830:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2712.014185] Lustre: server umount lustre-OST0000 complete [ 2712.018431] LustreError: 100830:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2720.792088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.996669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2721.001927] Lustre: Found index 0 for lustre-OST0000, updating log [ 2721.007912] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2721.046583] Lustre: lustre-OST0000: new disk, initializing [ 2721.050519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2721.055637] Lustre: 101362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2721.307542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.608892] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2731.110242] Lustre: Failing over lustre-OST0000 [ 2731.301465] Lustre: server umount lustre-OST0000 complete [ 2735.074632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.091060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.110790] Lustre: Skipped 1 previous similar message [ 2735.121962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.660489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.661357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.260437] Lustre: server umount lustre-MDT0000 complete [ 2743.777588] LustreError: 98281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.795323] LustreError: 98281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2744.695881] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650964 with bad export cookie 757835464931606937 [ 2744.709977] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2744.714157] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.203348] Lustre: server umount lustre-MDT0001 complete [ 2756.457388] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:56:15 (1786650975) [ 2772.653909] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2782.246691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.611550] LustreError: 103707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.767455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.114395] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2795.424798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.451927] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2803.627881] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2810.325035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.851667] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2824.625659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.751565] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2824.824305] Lustre: lustre-OST0001: new disk, initializing [ 2824.827415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2824.831760] Lustre: 106025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2825.160879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.171129] Lustre: Skipped 2 previous similar messages [ 2831.334716] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2832.419796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2832.428944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2832.539492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2839.432481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2850.505580] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2857.447711] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2857.462326] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.506257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2857.516251] Lustre: Skipped 1 previous similar message [ 2862.406213] Lustre: server umount lustre-MDT0001 complete [ 2863.087958] LustreError: 103704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.124049] LustreError: 103704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2875.069317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.206095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2875.217966] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2875.283412] Lustre: 103697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2876.987617] LustreError: 107555:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2877.003590] LustreError: 107555:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2877.027049] LustreError: 103699:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2877.259408] LustreError: 107570:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.139@tcp: cfg command failed: rc = -17 [ 2877.271488] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2877.271488] [ 2882.029667] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2882.817166] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2894.947757] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:58:33 (1786651113) [ 2914.062528] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2934.827506] Lustre: 109643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2957.887174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2960.967780] Lustre: 110778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3049.400778] Lustre: server umount lustre-MDT0000 complete [ 3049.956085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3049.973191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.999395] Lustre: Skipped 2 previous similar messages [ 3050.019585] LustreError: 104441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.045629] LustreError: 104441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3053.805768] LustreError: 103688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651273 with bad export cookie 757835464931607854 [ 3053.807049] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.827746] LustreError: 103688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3054.461582] Lustre: server umount lustre-MDT0001 complete [ 3070.160474] Lustre: server umount lustre-OST0000 complete [ 3072.294393] Lustre: 93316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786651276/real 1786651276] req@ffff8f3c8a7b2d80 x1873439055254400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786651292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3072.331579] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.345315] Lustre: Skipped 3 previous similar messages [ 3073.993374] Lustre: 93317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786651278/real 1786651278] req@ffff8f3c8a7b2300 x1873439055254656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786651294 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3074.253939] Lustre: server umount lustre-OST0001 complete [ 3080.024613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3089.534292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3097.749146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3108.201635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3127.824493] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3138.487578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.532031] Lustre: lustre-MDT0000: reset Object Index mappings [ 3139.314772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.322314] Lustre: Skipped 1 previous similar message [ 3145.903915] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3157.906437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.929017] Lustre: lustre-MDT0001: reset Object Index mappings [ 3164.196690] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3167.848874] Lustre: 115891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3175.623197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.682610] LustreError: 116245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.711827] LustreError: 116245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3183.729035] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3183.846700] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3184.715917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3196.099714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.040921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3202.064148] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3205.016152] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3214.701481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3218.660568] Lustre: 117768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3270.269852] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:04:49 (1786651489) [ 3296.739686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3296.752368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.794463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3296.800029] Lustre: Skipped 3 previous similar messages [ 3302.593712] Lustre: server umount lustre-MDT0000 complete [ 3306.593058] LustreError: 114722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651526 with bad export cookie 757835464931654754 [ 3306.614060] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.543862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3309.556328] Lustre: Skipped 5 previous similar messages [ 3313.296406] Lustre: server umount lustre-MDT0001 complete [ 3332.064443] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3332.275691] Lustre: server umount lustre-OST0000 complete [ 3343.078079] Lustre: server umount lustre-OST0001 complete [ 3352.477514] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 3367.300264] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3413.017344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.290684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3413.335718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3413.452776] Lustre: lustre-MDT0000: new disk, initializing [ 3413.616165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.625881] Lustre: Skipped 3 previous similar messages [ 3413.637745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3418.464612] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3425.817485] Lustre: server umount lustre-MDT0000 complete [ 3437.704114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.507587] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3438.535848] Lustre: 122053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3438.560568] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3445.257475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3453.475860] Lustre: server umount lustre-MDT0000 complete [ 3464.947968] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:08:03 (1786651683) [ 3466.739849] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3469.013817] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:08:07 (1786651687) [ 3487.931452] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3498.895749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.665659] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3513.818851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.946549] Lustre: 124104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3513.997626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3514.001194] Lustre: Skipped 1 previous similar message [ 3514.187575] Lustre: lustre-MDT0001: new disk, initializing [ 3514.408985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3514.437797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3520.023293] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3525.209554] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3533.305055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.826569] Lustre: lustre-OST0000: new disk, initializing [ 3533.842599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3533.854421] Lustre: 125682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3535.517720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3535.547098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3535.732093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3543.086086] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3556.409289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.641167] Lustre: lustre-OST0001: new disk, initializing [ 3556.643982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3556.648543] Lustre: 126553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3558.393523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3558.410949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3558.589677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3563.628748] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3572.807664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3576.637592] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3953.847463] Lustre: 132968:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3953.855121] Lustre: 132968:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3953.860369] Lustre: 132968:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3953.867914] Lustre: 132968:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3953.881093] Lustre: 132968:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3953.894369] Lustre: 132968:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4229.236275] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:20:47 (1786652447) [ 4246.500523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4246.508065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.533823] Lustre: Skipped 5 previous similar messages [ 4246.541417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.550190] Lustre: Skipped 2 previous similar messages [ 4249.576821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.245463] Lustre: server umount lustre-MDT0000 complete [ 4254.693271] LustreError: 124114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.709396] LustreError: 124114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4255.432847] LustreError: 125680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652475 with bad export cookie 757835464931721751 [ 4255.436972] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4255.454471] LustreError: 125680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4259.810452] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.814841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4259.819123] Lustre: Skipped 4 previous similar messages [ 4259.831562] Lustre: Skipped 4 previous similar messages [ 4262.355536] Lustre: server umount lustre-MDT0001 complete [ 4272.684052] Lustre: server umount lustre-OST0000 complete [ 4282.507557] Lustre: server umount lustre-OST0001 complete [ 4289.312369] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4299.928527] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4336.808646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.120455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.154651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4337.279809] Lustre: lustre-MDT0000: new disk, initializing [ 4337.466542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.486566] Lustre: Skipped 5 previous similar messages [ 4337.524745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4343.553598] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4355.317771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.492871] Lustre: 136017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4355.579422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4355.585778] Lustre: Skipped 1 previous similar message [ 4355.673909] Lustre: lustre-MDT0001: new disk, initializing [ 4355.869700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4355.890929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4361.844055] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4372.139861] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4379.264810] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4387.481678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.725240] Lustre: lustre-OST0000: new disk, initializing [ 4387.733849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4387.740423] Lustre: 137997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4387.858258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.862767] Lustre: Skipped 1 previous similar message [ 4389.691632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4389.721802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4389.811488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4395.581035] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4405.566819] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4416.776248] Lustre: lustre-MDD0000: changelog on [ 4419.614244] Lustre: lustre-MDD0001: changelog on [ 6110.151951] Lustre: lustre-MDD0001: changelog off [ 6111.908706] Lustre: 139630:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6112.723356] Lustre: lustre-MDD0000: changelog off [ 6119.905757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6119.914947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.930763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.947151] Lustre: Skipped 2 previous similar messages [ 6125.539161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.542143] Lustre: Skipped 2 previous similar messages [ 6130.667122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6130.678623] Lustre: Skipped 1 previous similar message [ 6132.192154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6132.437665] Lustre: server umount lustre-OST0000 complete [ 6135.777425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.778788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.808626] Lustre: Skipped 2 previous similar messages [ 6135.827276] Lustre: Skipped 1 previous similar message [ 6146.019503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.041925] Lustre: Skipped 3 previous similar messages [ 6150.115511] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6150.431978] Lustre: server umount lustre-MDT0000 complete [ 6151.140620] LustreError: 137592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.156174] LustreError: 137592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6154.958227] LustreError: 136008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654375 with bad export cookie 757835464932543418 [ 6154.959889] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6154.971395] LustreError: 136008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6155.667672] Lustre: server umount lustre-MDT0001 complete [ 6168.231621] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 6171.108485] Key type lgssc unregistered [ 6171.370352] LNet: 147099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6171.376259] LNetError: 147099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6171.390893] LNet: Removed LNI 192.168.201.139@tcp [ 6172.134166] Key type .llcrypt unregistered [ 6172.135902] Key type ._llcrypt unregistered [ 6191.375952] Key type ._llcrypt registered [ 6191.378318] Key type .llcrypt registered [ 6191.474375] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6207.092186] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6208.292235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6208.358274] alg: No test for adler32 (adler32-zlib) [ 6209.463070] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6209.890730] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 6211.624373] Key type lgssc registered [ 6212.834737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6249.449090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6249.483415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.698563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6250.726680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6250.782372] Lustre: lustre-MDT0000: new disk, initializing [ 6250.858506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.878638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6255.450904] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6268.583920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.734910] Lustre: 150318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6268.768889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6268.771784] Lustre: Skipped 1 previous similar message [ 6268.818944] Lustre: lustre-MDT0001: new disk, initializing [ 6268.886633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6268.925804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6268.934809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6273.381819] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6284.136563] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6290.368547] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6297.817775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.236275] Lustre: lustre-OST0000: new disk, initializing [ 6298.242378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6298.259787] Lustre: 152285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6298.382584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.065768] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6307.883462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6307.894326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6308.076945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6314.695576] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.952238] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_module obdecho/obdecho [ 6324.581473] LustreError: 153288:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6324.588195] LustreError: 153288:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6326.565568] LustreError: 153385:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6339.059552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.069487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.073787] Lustre: Skipped 1 previous similar message [ 6339.091669] Lustre: Skipped 2 previous similar messages [ 6344.165852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6344.170921] Lustre: Skipped 2 previous similar messages [ 6344.862550] Lustre: server umount lustre-MDT0000 complete [ 6348.242067] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654568 with bad export cookie 1422489731483207854 [ 6348.247623] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.253566] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6348.692143] Lustre: server umount lustre-MDT0001 complete [ 6364.116914] Lustre: server umount lustre-OST0000 complete [ 6386.606990] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6398.991964] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6436.235754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.453246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6436.506692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6436.655067] Lustre: lustre-MDT0000: new disk, initializing [ 6436.743749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6436.764873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6441.789810] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6453.659560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.777392] Lustre: 157103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6453.848622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6453.852240] Lustre: Skipped 1 previous similar message [ 6453.913874] Lustre: lustre-MDT0001: new disk, initializing [ 6453.991318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.037577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6454.053446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.273293] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6469.375691] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6476.131463] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6482.933118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.347472] Lustre: lustre-OST0000: new disk, initializing [ 6483.350989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6483.360782] Lustre: 159069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6483.438081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.671894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6488.688711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6488.850869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6490.680327] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6500.110682] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6505.239204] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6505.428535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6510.096866] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6510.358181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6514.668558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6514.676250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6514.677311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.684271] LustreError: Skipped 1 previous similar message [ 6514.733712] Lustre: Skipped 2 previous similar messages [ 6518.441418] Lustre: server umount lustre-OST0000 complete [ 6524.899711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.916643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6524.930632] Lustre: Skipped 2 previous similar messages [ 6529.011113] Lustre: server umount lustre-MDT0000 complete [ 6530.017609] LustreError: 157108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.037556] LustreError: 157108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6533.123043] LustreError: 157094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654753 with bad export cookie 1422489731483208813 [ 6533.140366] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.144469] LustreError: 157094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6533.662282] Lustre: server umount lustre-MDT0001 complete [ 6537.600511] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:59:16 (1786654756) [ 6539.536676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6541.703261] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:59:20 (1786654760) [ 6550.543408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.055640] LustreError: 161481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.128590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.142287] LustreError: 161482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.715439] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6561.290459] LustreError: 161481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6565.933399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.670087] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6580.117942] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6586.642355] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.481468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.948092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.960849] Lustre: Skipped 1 previous similar message [ 6600.576688] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6607.991607] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6619.116887] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 6620.567885] Lustre: server umount lustre-MDT0000 complete [ 6621.672994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.676417] LustreError: 161482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.703533] Lustre: Skipped 2 previous similar messages [ 6624.635758] LustreError: 161461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654844 with bad export cookie 1422489731483209695 [ 6624.641878] LustreError: 161461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6624.655112] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.786722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.798919] Lustre: Skipped 1 previous similar message [ 6629.370925] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 6629.379104] Lustre: Skipped 1 previous similar message [ 6629.793634] LustreError: 161477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.820088] LustreError: 161477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6631.262951] Lustre: server umount lustre-MDT0001 complete [ 6637.778781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6652.487880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.826945] LustreError: 165334:0:(llog.c:1655:llog_backup()) MGC192.168.201.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6652.837414] Lustre: 165334:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6654.512720] LustreError: 165339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.745750] LustreError: 165339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.757631] LustreError: 165339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6713.824298] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6713.833681] Lustre: 165334:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6754.274304] LustreError: 165341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6754.285237] LustreError: 165341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 6754.848632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6759.881245] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6759.918505] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6768.908417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.123148] LustreError: 148538:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f17ff3af x1873442954672896/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6773.623579] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6774.778978] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6781.135674] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6782.764803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6788.425772] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6789.874687] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6808.545331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.548939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6808.556178] Lustre: Skipped 2 previous similar messages [ 6808.563060] Lustre: Skipped 3 previous similar messages [ 6813.578597] Lustre: server umount lustre-MDT0000 complete [ 6818.591417] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655038 with bad export cookie 1422489731483216149 [ 6818.615042] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.629953] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6819.213487] Lustre: server umount lustre-MDT0001 complete [ 6825.784774] Lustre: server umount lustre-OST0000 complete [ 6850.049800] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6862.239705] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6901.231435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.489868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6901.516220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6901.615821] Lustre: lustre-MDT0000: new disk, initializing [ 6901.691197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.694237] Lustre: Skipped 1 previous similar message [ 6901.738960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6907.214458] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6919.968283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.169135] Lustre: 170445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6920.664194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6926.226256] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6936.065111] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6942.917577] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6950.309773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.578187] Lustre: lustre-OST0000: new disk, initializing [ 6950.581208] Lustre: Skipped 1 previous similar message [ 6950.584788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6950.592662] Lustre: Skipped 2 previous similar messages [ 6950.597870] Lustre: 172415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6957.266957] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6959.233712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6959.243123] Lustre: Skipped 1 previous similar message [ 6959.252353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6959.349716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6966.909518] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6971.454644] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6971.667588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6976.914656] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6977.114246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6980.067730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6980.067747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.069071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6980.069078] Lustre: Skipped 1 previous similar message [ 6980.082064] LustreError: Skipped 1 previous similar message [ 6980.116513] Lustre: Skipped 2 previous similar messages [ 6992.870689] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.221543] Lustre: server umount lustre-OST0000 complete [ 7012.320165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7012.694336] Lustre: server umount lustre-MDT0000 complete [ 7015.909351] LustreError: 170455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.938150] LustreError: 170455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7017.012452] LustreError: 170435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655237 with bad export cookie 1422489731483232984 [ 7017.015042] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.023909] LustreError: 170435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7017.613830] Lustre: server umount lustre-MDT0001 complete [ 7028.144530] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:07:27 (1786655247) [ 7034.690537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.188531] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7050.247636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.721577] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7066.625576] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7073.139229] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7080.725045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.470178] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7095.260499] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.422078] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:08:46 (1786655326) [ 7116.770392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7116.776050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.792983] LustreError: Skipped 1 previous similar message [ 7116.806869] Lustre: Skipped 2 previous similar messages [ 7116.813338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7116.833656] Lustre: Skipped 12 previous similar messages [ 7119.240108] Lustre: server umount lustre-OST0000 complete [ 7122.913589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7137.248440] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7137.602268] Lustre: server umount lustre-MDT0000 complete [ 7142.064268] LustreError: 174809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655362 with bad export cookie 1422489731483233866 [ 7142.082395] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.557583] Lustre: server umount lustre-MDT0001 complete [ 7153.599756] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7156.742521] Key type lgssc unregistered [ 7157.027304] LNet: 178561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7157.038107] LNetError: 178561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.062355] LNet: Removed LNI 192.168.201.139@tcp [ 7158.143510] Key type .llcrypt unregistered [ 7158.151073] Key type ._llcrypt unregistered [ 7179.873738] Key type ._llcrypt registered [ 7179.876810] Key type .llcrypt registered [ 7180.027770] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7181.756928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.838118] alg: No test for adler32 (adler32-zlib) [ 7183.232905] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7183.691070] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7185.480260] Key type lgssc registered [ 7187.078905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7195.123814] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7195.148590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.720362] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7215.841477] LustreError: 179806:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [ 7220.960764] LustreError: 179806:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7220.971200] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7220.980289] LustreError: MGC192.168.201.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7220.994381] LustreError: 179806:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7221.004523] LustreError: 179806:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7221.015376] LustreError: 179806:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7221.077982] Lustre: server umount lustre-OST0000 complete [ 7221.082389] LustreError: 179806:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7229.800082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.322405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.301322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7241.699638] LustreError: 180305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.709639] LustreError: 180305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7245.262954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.589092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7250.252655] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7258.467659] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7264.763216] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7271.384452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7271.580585] LustreError: 182121:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7271.600393] Lustre: 182121:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7271.784481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.932854] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7286.242831] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.403685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.409382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.435210] Lustre: Skipped 2 previous similar messages [ 7290.463588] Lustre: Skipped 2 previous similar messages [ 7294.766210] Lustre: server umount lustre-MDT0000 complete [ 7295.458526] LustreError: 181313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.469682] LustreError: 181313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7297.791408] LustreError: 180287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655517 with bad export cookie 3754507606183381760 [ 7297.816923] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7298.722102] LustreError: 181034:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000042cabd75 x1873443975884288/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7298.914146] Lustre: server umount lustre-MDT0001 complete [ 7313.564962] Lustre: server umount lustre-OST0000 complete [ 7322.716628] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7332.169331] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7363.793923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.042896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7364.078427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7364.158923] Lustre: lustre-MDT0000: new disk, initializing [ 7364.281588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.320323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7369.433829] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7380.672145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.815716] Lustre: 185644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7380.873658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7380.877669] Lustre: Skipped 1 previous similar message [ 7380.959740] Lustre: lustre-MDT0001: new disk, initializing [ 7381.062332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7381.103296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7381.124968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7385.680204] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7395.513768] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7400.955888] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.773172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.985648] Lustre: lustre-OST0000: new disk, initializing [ 7407.988826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7408.012875] Lustre: 187611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7408.101979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.832835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7413.860050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7414.039501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7414.581856] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7423.965506] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.029307] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7428.194683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7432.220614] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7432.405250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7434.730817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7434.736644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7434.746817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.754716] LustreError: Skipped 1 previous similar message [ 7434.771618] Lustre: Skipped 1 previous similar message [ 7439.846449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7439.859995] Lustre: Skipped 1 previous similar message [ 7440.730665] Lustre: server umount lustre-OST0000 complete [ 7444.961487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.977575] Lustre: Skipped 1 previous similar message [ 7444.992697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7445.002790] Lustre: Skipped 1 previous similar message [ 7450.087124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.093682] Lustre: Skipped 1 previous similar message [ 7450.822344] Lustre: server umount lustre-MDT0000 complete [ 7454.943467] LustreError: 185634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655674 with bad export cookie 3754507606183382481 [ 7454.944923] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.978060] LustreError: 185634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7455.361225] Lustre: server umount lustre-MDT0001 complete [ 7464.757743] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:14:43 (1786655683) [ 7480.970298] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7483.322457] Key type lgssc unregistered [ 7483.555166] LNet: 190561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7483.558767] LNetError: 190561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7483.570440] LNet: Removed LNI 192.168.201.139@tcp [ 7484.368706] Key type .llcrypt unregistered [ 7484.371678] Key type ._llcrypt unregistered [ 7500.395777] Key type ._llcrypt registered [ 7500.397518] Key type .llcrypt registered [ 7500.534931] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7501.437867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7501.473521] alg: No test for adler32 (adler32-zlib) [ 7502.661481] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7502.840299] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7504.496155] Key type lgssc registered [ 7505.605778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7513.456354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7513.471758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.133828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.113805] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7525.348584] LustreError: 191779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.387527] LustreError: 191779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7528.318504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.791768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7533.304262] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7540.880570] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.260785] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.897569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.105595] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7555.115439] LustreError: 193645:0:(llog.c:1696:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7555.141768] Lustre: 193645:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7555.247438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.059345] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7570.851959] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.793208] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7573.808414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.844292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7575.012226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7575.034739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7575.053821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7579.111902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7579.123335] Lustre: Skipped 1 previous similar message [ 7579.124425] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.407882] Lustre: server umount lustre-MDT0000 complete [ 7584.226954] LustreError: 191779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.240608] LustreError: 191779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7584.923316] LustreError: 191759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655804 with bad export cookie 14812593405012292762 [ 7584.926673] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7584.937757] LustreError: 191759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7586.144407] LustreError: 192510:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cffba6d9 x1873444310381696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7586.426622] Lustre: server umount lustre-MDT0001 complete [ 7600.240285] Lustre: server umount lustre-OST0000 complete [ 7612.031789] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7625.225117] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7664.179818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.671770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.694077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7664.776940] Lustre: lustre-MDT0000: new disk, initializing [ 7664.836563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.852268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7670.098700] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7682.808158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.903711] Lustre: 197172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.941626] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7682.944683] Lustre: Skipped 1 previous similar message [ 7682.947982] LustreError: 197955:0:(llog.c:1696:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7682.953858] LustreError: 197955:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7682.957820] Lustre: 197955:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7682.974457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7682.982269] Lustre: Skipped 1 previous similar message [ 7683.056913] Lustre: lustre-MDT0001: new disk, initializing [ 7683.214191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7683.277264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7683.297852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7688.273396] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7700.174206] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7705.847776] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7713.311447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.580857] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7713.590182] Lustre: Skipped 1 previous similar message [ 7713.596550] LustreError: 199118:0:(llog.c:1696:llog_backup()) MGC192.168.201.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7713.611989] LustreError: 199118:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7713.627445] Lustre: 199118:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7713.635976] Lustre: 199118:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7713.687099] Lustre: lustre-OST0000: new disk, initializing [ 7713.690977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7713.697722] Lustre: 199141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7713.815305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.891341] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7723.111767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7723.141985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7723.274033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7731.107633] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.079653] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7736.280577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7741.849326] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7742.272946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7749.097768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.098913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7749.106810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7749.121933] Lustre: Skipped 1 previous similar message [ 7749.149808] Lustre: Skipped 2 previous similar messages [ 7750.974433] Lustre: server umount lustre-OST0000 complete [ 7759.331583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.354872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7762.613934] Lustre: server umount lustre-MDT0000 complete [ 7764.457653] LustreError: 197180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7764.487198] LustreError: 197180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7766.923676] LustreError: 197967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655986 with bad export cookie 14812593405012293483 [ 7766.933932] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7767.258905] Lustre: server umount lustre-MDT0001 complete [ 7777.926481] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:19:56 (1786655996) [ 7795.174246] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7806.298754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.671169] LustreError: 202269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.727810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7812.086193] LustreError: 202270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.293935] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7821.294123] LustreError: 202269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7821.323759] LustreError: 202269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7822.130748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.701692] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7830.920119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7839.045894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.685277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.695331] Lustre: Skipped 1 previous similar message [ 7846.204217] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7855.365142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.492930] Lustre: lustre-OST0001: new disk, initializing [ 7855.497276] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7855.504225] Lustre: 204576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7861.281244] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7864.425592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7864.439169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7864.589698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7871.330489] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7877.244053] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7878.897940] Lustre: DEBUG MARKER: ADD OST3 [ 7883.500380] loop: module loaded [ 7883.567591] loop0: detected capacity change from 0 to 204800000 [ 7883.791240] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7883.802774] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7883.815697] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7889.721358] loop0: detected capacity change from 0 to 204800000 [ 7891.193760] Lustre: DEBUG MARKER: STOP OST3 [ 7896.592306] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7896.810162] Lustre: lustre-OST0003: new disk, initializing [ 7896.818337] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7896.826547] Lustre: 206563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.824303] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7899.030362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7899.040542] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7899.070103] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7900.832078] LustreError: 206759:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7901.280351] Lustre: Failing over lustre-OST0003 [ 7901.842633] Lustre: server umount lustre-OST0003 complete [ 7904.254470] LustreError: 205061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.256519] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.258336] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.258343] Lustre: Skipped 1 previous similar message [ 7904.271104] LustreError: 205061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7904.339326] LustreError: Skipped 1 previous similar message [ 7904.785304] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7913.864438] Lustre: DEBUG MARKER: START OST3 again [ 7921.470260] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7922.207257] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7922.210606] Lustre: Skipped 2 previous similar messages [ 7922.224120] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7923.633895] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7923.680696] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7923.681694] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7929.596457] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7950.541348] Lustre: Failing over lustre-OST0003 [ 7950.620914] Lustre: server umount lustre-OST0003 complete [ 7952.926772] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.944788] Lustre: Skipped 1 previous similar message [ 7952.955315] LustreError: 203756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.979729] LustreError: 203756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7968.232637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7968.243774] Lustre: Skipped 3 previous similar messages [ 7974.930233] Lustre: server umount lustre-MDT0000 complete [ 7979.206486] LustreError: 202249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656199 with bad export cookie 14812593405012294365 [ 7979.208845] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7979.223045] LustreError: 202249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7984.616394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7984.633907] Lustre: Skipped 7 previous similar messages [ 7985.811626] Lustre: server umount lustre-MDT0001 complete [ 7990.011730] Lustre: server umount lustre-OST0000 complete [ 7994.204462] Lustre: server umount lustre-OST0001 complete [ 8001.520705] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8010.425402] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8056.153674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.377908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8056.418546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8056.554924] Lustre: lustre-MDT0000: new disk, initializing [ 8056.669426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.689039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8061.352730] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8072.357259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.498745] Lustre: 212034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8072.569734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8072.573758] Lustre: Skipped 1 previous similar message [ 8072.653808] Lustre: lustre-MDT0001: new disk, initializing [ 8072.834403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8072.855822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8077.459679] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8087.451351] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8093.956831] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8101.229622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.485033] Lustre: 214003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8103.282812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8103.388712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8107.892034] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8117.795471] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8123.550413] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8123.767650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8127.835163] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8128.033207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8134.126952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.147190] Lustre: Skipped 7 previous similar messages [ 8134.166901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.170517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.180491] Lustre: Skipped 1 previous similar message [ 8136.386649] Lustre: server umount lustre-OST0000 complete [ 8144.359158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8147.056355] Lustre: server umount lustre-MDT0000 complete [ 8149.478377] LustreError: 212046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8149.509761] LustreError: 212046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8151.811172] LustreError: 212027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656371 with bad export cookie 14812593405012296661 [ 8151.828820] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.831880] LustreError: 212027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8152.389755] Lustre: server umount lustre-MDT0001 complete [ 8156.252913] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:26:15 (1786656375) [ 8175.408901] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8186.681893] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8228.167270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8228.527567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8228.555760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8228.561464] Lustre: Skipped 1 previous similar message [ 8228.615973] Lustre: lustre-MDT0000: new disk, initializing [ 8228.617974] Lustre: Skipped 1 previous similar message [ 8228.687882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8228.699995] Lustre: Skipped 1 previous similar message [ 8233.626477] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8245.965367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.182762] Lustre: 218617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8246.661401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8252.151665] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8262.583175] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8269.666808] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8277.590783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8277.856038] Lustre: 220584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8279.346237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8279.522834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8285.275324] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8296.853377] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.871934] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8303.162798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8308.559770] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8308.812572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8315.364720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.372392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.384059] LustreError: Skipped 1 previous similar message [ 8315.391499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.391507] Lustre: Skipped 2 previous similar messages [ 8315.435275] Lustre: Skipped 4 previous similar messages [ 8317.126400] Lustre: server umount lustre-OST0000 complete [ 8327.634570] Lustre: server umount lustre-MDT0000 complete [ 8330.729643] LustreError: 218627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.749777] LustreError: 218627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8332.614089] LustreError: 218607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656552 with bad export cookie 14812593405012297543 [ 8332.635061] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.647868] LustreError: 218607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8333.207445] Lustre: server umount lustre-MDT0001 complete [ 8346.411154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.265719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.269025] Lustre: Skipped 5 previous similar messages [ 8352.569592] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8362.127583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.678461] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8378.347335] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8385.086560] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8395.110837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.365097] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8411.993686] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8431.079260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8431.082611] LustreError: Skipped 1 previous similar message [ 8434.508550] Lustre: server umount lustre-OST0000 complete [ 8445.189142] Lustre: server umount lustre-MDT0000 complete [ 8449.618397] LustreError: 225701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656669 with bad export cookie 14812593405012298425 [ 8449.632038] LustreError: 225701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8449.632421] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8450.184422] Lustre: server umount lustre-MDT0001 complete [ 8551.390322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8554.024630] Key type lgssc unregistered [ 8554.353739] LNet: 226489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.359762] LNetError: 226489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.381454] LNet: Removed LNI 192.168.201.139@tcp [ 8555.132266] Key type .llcrypt unregistered [ 8555.134867] Key type ._llcrypt unregistered [ 8581.131546] Key type ._llcrypt registered [ 8581.133850] Key type .llcrypt registered [ 8581.330786] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8600.857365] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8603.135234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8603.285959] alg: No test for adler32 (adler32-zlib) [ 8604.456535] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8604.916537] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8606.699576] Key type lgssc registered [ 8608.538166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8646.146571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8646.187904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8647.369464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8647.405826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8647.487586] Lustre: lustre-MDT0000: new disk, initializing [ 8647.595454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.621797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8652.362778] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8665.840774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.076938] Lustre: 229844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8666.232516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8666.240738] Lustre: Skipped 1 previous similar message [ 8666.408916] Lustre: lustre-MDT0001: new disk, initializing [ 8666.603315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8666.780630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8666.808967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8672.268048] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8685.634872] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8692.924728] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8701.637904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.026849] Lustre: lustre-OST0000: new disk, initializing [ 8702.031211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8702.037311] Lustre: 231813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8702.178657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.705211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8707.727177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8707.969846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8709.741466] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8722.428638] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8728.988553] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8729.183387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8734.530724] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8734.720783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8738.796965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.803762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8738.821784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8738.830194] Lustre: Skipped 1 previous similar message [ 8738.861053] LustreError: Skipped 1 previous similar message [ 8742.862621] Lustre: server umount lustre-OST0000 complete [ 8748.515112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8748.524479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.541360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8748.548796] Lustre: Skipped 1 previous similar message [ 8753.818131] Lustre: server umount lustre-MDT0000 complete [ 8758.247838] LustreError: 230640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656978 with bad export cookie 3274407241835244591 [ 8758.254126] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8758.261261] LustreError: 230640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8758.507666] Lustre: server umount lustre-MDT0001 complete [ 8778.953402] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8791.319960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.904506] LustreError: 234751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.922106] LustreError: 234751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8791.996191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.157962] LustreError: 234752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8798.005744] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8801.266125] LustreError: 234751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.375799] LustreError: 234752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.560730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.190216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8814.141356] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8817.475181] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8825.189330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8825.689430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.720977] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8842.102730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8842.259572] Lustre: lustre-OST0001: new disk, initializing [ 8842.262453] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8842.270244] Lustre: 237060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8842.349613] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8848.644806] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8851.618035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8851.636631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8851.830105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8860.364967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8866.543893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8879.520331] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:38:18 (1786657098) [ 8882.672082] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.677821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.692630] Lustre: Skipped 3 previous similar messages [ 8882.700860] Lustre: Skipped 3 previous similar messages [ 8887.782062] LustreError: 234746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8887.804526] LustreError: 234746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8887.838144] Lustre: server umount lustre-MDT0000 complete [ 8892.377439] LustreError: 237059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657112 with bad export cookie 3274407241835245473 [ 8892.395658] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.878745] Lustre: server umount lustre-MDT0001 complete [ 8907.204782] Lustre: server umount lustre-OST0000 complete [ 8908.256085] Lustre: 228066:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786657112/real 1786657112] req@ffff8f3c8345f800 x1873445465996032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786657128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8908.281573] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.296923] Lustre: Skipped 1 previous similar message [ 8912.225642] Lustre: server umount lustre-OST0001 complete [ 8918.994283] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 8929.164948] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8970.569281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.996244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8971.094835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8971.273632] Lustre: lustre-MDT0000: new disk, initializing [ 8971.477616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8971.532286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8976.864561] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8991.151144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.348602] Lustre: 241008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8991.419603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8991.428235] Lustre: Skipped 1 previous similar message [ 8991.531533] Lustre: lustre-MDT0001: new disk, initializing [ 8991.742148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8991.753191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8997.493590] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9007.704337] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9013.259395] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9020.212408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.447354] Lustre: lustre-OST0000: new disk, initializing [ 9020.451957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9020.458638] Lustre: 242977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9021.996726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9022.006939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9022.166355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9027.084714] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9036.660272] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.103678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9042.339535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9047.524958] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9047.794764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9053.165515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9053.166528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.181727] LustreError: Skipped 1 previous similar message [ 9053.194180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9053.194191] Lustre: Skipped 2 previous similar messages [ 9053.226074] Lustre: Skipped 1 previous similar message [ 9056.397738] Lustre: server umount lustre-OST0000 complete [ 9063.392974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9063.396260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9063.442071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9063.447524] Lustre: Skipped 2 previous similar messages [ 9067.220732] Lustre: server umount lustre-MDT0000 complete [ 9068.514333] LustreError: 241818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9068.535120] LustreError: 241818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9070.746656] LustreError: 241001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657290 with bad export cookie 3274407241835247608 [ 9070.747151] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9070.757751] LustreError: 241001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9071.301212] Lustre: server umount lustre-MDT0001 complete [ 9081.174779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.725738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.733663] Lustre: Skipped 2 previous similar messages [ 9086.322920] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9086.957022] LustreError: 245196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9086.979775] LustreError: 245196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9095.732828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.379642] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9110.511668] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9118.018562] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9126.308692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9135.208580] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9145.782558] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9293.793423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.799505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.812230] Lustre: Skipped 2 previous similar messages [ 9293.814135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9298.529459] Lustre: server umount lustre-OST0000 complete [ 9309.001951] Lustre: server umount lustre-MDT0000 complete [ 9309.159023] LustreError: 245197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.186285] LustreError: 245197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9312.672657] LustreError: 245177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657532 with bad export cookie 3274407241835248490 [ 9312.678216] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9312.907789] Lustre: server umount lustre-MDT0001 complete [ 9415.612521] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9418.254514] Key type lgssc unregistered [ 9418.459339] LNet: 248810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9418.465491] LNetError: 248810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9418.479084] LNet: Removed LNI 192.168.201.139@tcp [ 9419.360218] Key type .llcrypt unregistered [ 9419.362302] Key type ._llcrypt unregistered [ 9446.676893] Key type ._llcrypt registered [ 9446.679453] Key type .llcrypt registered [ 9446.802081] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9461.724442] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9462.624449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9462.729344] alg: No test for adler32 (adler32-zlib) [ 9463.902217] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9464.183726] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9465.832149] Key type lgssc registered [ 9466.939650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9515.833807] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9527.587044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.611118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.990281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9529.014979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9529.100820] Lustre: lustre-MDT0000: new disk, initializing [ 9529.208922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9529.223710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9534.426288] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9545.427416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9545.589141] Lustre: 252878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9545.647202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9545.650635] Lustre: Skipped 1 previous similar message [ 9545.710287] Lustre: lustre-MDT0001: new disk, initializing [ 9545.791525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9545.843136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9545.856209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9549.845862] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9554.488035] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9560.660941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.984488] Lustre: lustre-OST0000: new disk, initializing [ 9560.993809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9560.999895] Lustre: 254509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9561.128411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.816416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9561.837190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9561.970830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9567.829658] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9578.398946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.570466] Lustre: lustre-OST0001: new disk, initializing [ 9578.576267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9578.582422] Lustre: 255381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9578.721813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9584.337879] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9585.295258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9585.317179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9585.491923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9596.717428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9607.045165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9621.478996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9621.494028] Lustre: Skipped 2 previous similar messages [ 9621.799474] Lustre: server umount lustre-MDT0000 complete [ 9625.475644] LustreError: 252872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657845 with bad export cookie 8495708329587470851 [ 9625.478592] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.483638] LustreError: 252872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9625.750950] Lustre: server umount lustre-MDT0001 complete [ 9639.592433] Lustre: server umount lustre-OST0000 complete [ 9642.464148] Lustre: 250388:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786657846/real 1786657846] req@ffff8f3c8a58f480 x1873446367735936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786657862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9642.491438] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.508927] Lustre: Skipped 1 previous similar message [ 9642.791697] Lustre: server umount lustre-OST0001 complete [ 9653.749737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9661.263954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9667.917317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9675.636569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9690.576475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9699.834485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.871799] Lustre: lustre-MDT0000: reset Object Index mappings [ 9700.465502] LustreError: 260391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.477863] LustreError: 260391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9700.557500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9705.428806] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9705.958025] LustreError: 260392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.051249] LustreError: 260391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.500898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.535625] Lustre: lustre-MDT0001: reset Object Index mappings [ 9715.163447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9719.535731] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9722.282928] Lustre: 261534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9729.192584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.506330] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9740.914581] LustreError: 261890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.008026] LustreError: 262408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.029847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9745.042634] LustreError: 262408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9747.893514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.201098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9748.207306] Lustre: Skipped 1 previous similar message [ 9753.590709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9753.949293] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9761.663143] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9765.924197] Lustre: 263404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9771.490328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9771.501225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.529275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9773.775993] Lustre: server umount lustre-MDT0000 complete [ 9774.062403] LustreError: 260392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.067192] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.102623] Lustre: Skipped 2 previous similar messages [ 9778.090712] LustreError: 263407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657998 with bad export cookie 8495708329587477641 [ 9778.101227] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9778.102664] LustreError: 263407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9778.500313] Lustre: server umount lustre-MDT0001 complete [ 9792.805812] Lustre: server umount lustre-OST0000 complete [ 9795.616176] Lustre: 250387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786657999/real 1786657999] req@ffff8f3dabbcc700 x1873446367784576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786658015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9795.639071] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.866290] Lustre: server umount lustre-OST0001 complete [ 9817.744175] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:53:57 (1786658037) [ 9835.912054] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9845.437946] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9878.698937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.107824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9879.177262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9879.314482] Lustre: lustre-MDT0000: new disk, initializing [ 9879.370355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.388946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9884.733895] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9895.969906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.142849] Lustre: 267660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9896.256937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9896.263074] Lustre: Skipped 1 previous similar message [ 9896.364626] Lustre: lustre-MDT0001: new disk, initializing [ 9896.604645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9896.625789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9900.606463] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9908.227522] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9912.400110] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9918.126264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.351252] Lustre: lustre-OST0000: new disk, initializing [ 9918.355016] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9918.360143] Lustre: 269628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9920.325902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9920.333096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.476439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9923.950629] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9937.222337] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9944.715313] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9945.117369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9952.781849] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9952.940960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9956.322416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9956.336691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9956.342265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9961.451135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9961.461611] Lustre: Skipped 2 previous similar messages [ 9962.094556] Lustre: server umount lustre-OST0000 complete [ 9966.588336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9966.602797] Lustre: Skipped 1 previous similar message [ 9971.681880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9971.694984] Lustre: Skipped 1 previous similar message [ 9980.385284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9980.562166] Lustre: server umount lustre-MDT0000 complete [ 9981.931979] LustreError: 267671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.973147] LustreError: 267671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9983.636382] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658203 with bad export cookie 8495708329587479734 [ 9983.646785] LustreError: 267653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9983.648485] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9984.021596] Lustre: server umount lustre-MDT0001 complete [10005.200390] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10017.980652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.848836] LustreError: 272566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10018.979588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10018.985741] Lustre: Skipped 2 previous similar messages [10026.160153] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10035.147109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.724651] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10044.353781] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10053.709929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10062.277295] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10072.533766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.769247] Lustre: lustre-OST0001: new disk, initializing [10072.776317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10072.792433] Lustre: 274878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10074.191509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10074.222600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10074.373864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10078.879232] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10088.001221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10091.023660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10096.016936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10096.043654] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10101.681202] Lustre: server umount lustre-MDT0000 complete [10102.247369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10102.262556] LustreError: Skipped 1 previous similar message [10102.268985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.284601] Lustre: Skipped 3 previous similar messages [10102.289205] LustreError: 273552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.304158] LustreError: 273552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10104.638922] LustreError: 272548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658324 with bad export cookie 8495708329587480616 [10104.646571] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10104.648184] LustreError: 272548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10105.318151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10105.326059] Lustre: Skipped 3 previous similar messages [10107.072145] Lustre: server umount lustre-MDT0001 complete [10110.709410] Lustre: server umount lustre-OST0000 complete [10114.218383] Lustre: server umount lustre-OST0001 complete [10129.679763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10139.432609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.014539] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10152.461287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.801095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10152.815731] Lustre: Skipped 4 previous similar messages [10157.016841] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10159.399439] Lustre: 279068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10165.663940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10172.619342] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10181.365633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10184.800053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10184.800554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10187.691119] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10194.817869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10198.201529] Lustre: 280944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10203.059440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10203.204926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10212.314189] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:00:31 (1786658431) [10229.732690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10229.741152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10229.755755] Lustre: Skipped 4 previous similar messages [10229.761541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10233.173835] Lustre: server umount lustre-MDT0000 complete [10235.885575] LustreError: 277925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.897431] LustreError: 277925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10236.432432] LustreError: 277909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658456 with bad export cookie 8495708329587486909 [10236.449495] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.884906] Lustre: server umount lustre-MDT0001 complete [10252.343574] Lustre: server umount lustre-OST0000 complete [10261.846646] Lustre: server umount lustre-OST0001 complete [10270.068898] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [10279.444414] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10329.577775] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10339.987669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.156544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10340.204430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10340.358302] Lustre: lustre-MDT0000: new disk, initializing [10340.538247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10345.045530] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10358.394967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.576333] Lustre: 285197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10359.195363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10364.328102] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10378.743224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.986777] Lustre: 286832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10380.565406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10386.197420] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10395.832245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10396.020252] Lustre: 287705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10397.381543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10397.390033] Lustre: Skipped 1 previous similar message [10397.494573] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10402.249353] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10411.454304] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10415.212990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10415.218307] Lustre: Skipped 1 previous similar message [10430.815924] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:04:09 (1786658649) [10433.511938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10433.532260] Lustre: Skipped 7 previous similar messages [10433.540452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10433.552471] Lustre: Skipped 11 previous similar messages [10439.049972] Lustre: server umount lustre-MDT0000 complete [10443.093965] LustreError: 285189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658663 with bad export cookie 8495708329587494826 [10443.111575] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10443.114527] LustreError: 285189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10443.440761] Lustre: server umount lustre-MDT0001 complete [10457.649374] Lustre: server umount lustre-OST0000 complete [10459.110813] Lustre: 250387:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786658663/real 1786658663] req@ffff8f3d90305f80 x1873446368037248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786658679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10462.157509] Lustre: server umount lustre-OST0001 complete [10471.514915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.073149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10472.082208] Lustre: Skipped 6 previous similar messages [10476.963893] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10488.634636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.393549] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10505.035941] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10510.145075] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10516.791783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.970582] LustreError: 291778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10517.005956] LustreError: 291778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10524.197743] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10531.541607] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10551.900347] loop1: detected capacity change from 0 to 204800000 [10552.127689] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10552.134824] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10552.170551] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10557.003764] loop1: detected capacity change from 0 to 204800000 [10562.059456] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10562.140412] LustreError: 289937:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs40: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10562.156565] LustreError: 289937:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs40 [10562.162666] LustreError: 289937:0:(mgs_handler.c:517:mgs_target_reg()) fs40-MDT0001: Error checking target index: rc = -13 [10562.167250] LustreError: fs40-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10562.172540] LustreError: 293668:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10562.177263] LustreError: 293668:0:(tgt_mount.c:2129:server_put_super()) no obd fs40-MDT0001 [10562.180970] LustreError: 293668:0:(tgt_mount.c:116:server_deregister_mount()) fs40-MDT0001 not registered [10562.211917] Lustre: server umount fs40-MDT0001 complete [10562.214239] LustreError: 293668:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10570.472429] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10570.529441] Lustre: Setting parameter fs40-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs40-MDT0001 [10570.564739] Lustre: srv-fs40-MDT0001: No data found on store. Initialize space. [10570.570649] Lustre: Skipped 4 previous similar messages [10570.654923] Lustre: fs40-MDT0001: new disk, initializing [10570.665798] Lustre: Skipped 3 previous similar messages [10570.853894] Lustre: srv-fs40-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10570.872162] Lustre: 294134:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs40-MDT0001-mdtlov: can't create new llog: rc = -115 [10570.886355] LustreError: 294134:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs40-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10574.636709] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10582.572341] Lustre: server umount fs40-MDT0001 complete [10586.082468] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10597.686458] loop1: detected capacity change from 0 to 204800000 [10604.311653] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10604.712138] Lustre: 295702:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs40-MDT0001-mdtlov: can't create new llog: rc = -115 [10604.725654] LustreError: 295702:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs40-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10610.135678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10614.688620] Lustre: server umount fs40-MDT0001 complete [10616.858766] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10624.486779] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [10624.491523] Lustre: Skipped 4 previous similar messages [10627.044718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10627.080511] Lustre: Skipped 4 previous similar messages [10638.304393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10638.598644] Lustre: server umount lustre-MDT0000 complete [10648.324609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.550698] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.225292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10654.659708] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10665.890545] loop1: detected capacity change from 0 to 204800000 [10672.768875] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.113623] Lustre: 298151:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs40-MDT0001-mdtlov: can't create new llog: rc = -115 [10673.130604] LustreError: 298151:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs40-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10678.672174] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10689.579542] Lustre: server umount fs40-MDT0001 complete [10718.181610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10722.782300] Lustre: server umount lustre-OST0000 complete [10734.655255] Lustre: server umount lustre-MDT0000 complete [10739.955302] LustreError: 289931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658959 with bad export cookie 8495708329587497507 [10739.968182] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10740.552846] Lustre: server umount lustre-MDT0001 complete [10752.909593] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10756.076565] Key type lgssc unregistered [10756.434786] LNet: 300504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10756.446636] LNetError: 300504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10756.472316] LNet: Removed LNI 192.168.201.139@tcp [10757.674422] Key type .llcrypt unregistered [10757.685869] Key type ._llcrypt unregistered [10784.704550] Key type ._llcrypt registered [10784.710280] Key type .llcrypt registered [10784.937717] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10786.076869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10786.234558] alg: No test for adler32 (adler32-zlib) [10787.736468] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [10788.111757] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10789.817049] Key type lgssc registered [10790.937898] Lustre: Echo OBD driver; http://www.lustre.org/ [10799.177548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10799.200860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.886593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10806.710480] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10810.345355] LustreError: 301776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.379584] LustreError: 301776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10815.465026] LustreError: 301777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.963857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.681197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10822.620142] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10830.861564] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10836.663813] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10844.725511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10845.392929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10846.374991] LustreError: 303608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.824529] LustreError: 303609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.509875] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10856.948203] LustreError: 303662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.055974] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10867.172040] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.195958] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10880.174547] loop1: detected capacity change from 0 to 204800000 [10880.318388] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10880.329757] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10880.349945] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.585134] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.612213] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10885.749480] loop1: detected capacity change from 0 to 204800000 [10892.123372] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.413660] Lustre: fs20676-OST0000: new disk, initializing [10892.416177] Lustre: srv-fs20676-OST0000: No data found on store. Initialize space. [10892.437880] Lustre: 305512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10892.505914] Lustre: fs20676-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10899.622765] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10909.003681] Lustre: server umount fs20676-OST0000 complete [10918.378879] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.430920] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10923.651780] loop1: detected capacity change from 0 to 204800000 [10928.611633] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10930.853855] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.045538] Lustre: fs20676-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10937.268916] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10943.990167] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10945.908184] loop2: detected capacity change from 0 to 204800000 [10946.260985] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10946.271836] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10946.280839] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10952.127729] loop2: detected capacity change from 0 to 204800000 [10961.324706] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.487634] LustreError: fs20676-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10961.496747] LustreError: fs20676-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10961.504503] LustreError: 308554:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10961.509569] LustreError: 308554:0:(tgt_mount.c:2129:server_put_super()) no obd fs20676-OST0001 [10961.514158] LustreError: 308554:0:(tgt_mount.c:116:server_deregister_mount()) fs20676-OST0001 not registered [10961.544253] Lustre: server umount fs20676-OST0001 complete [10961.546898] LustreError: 308554:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10975.281447] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10975.537600] Lustre: fs20676-OST0001: new disk, initializing [10975.541237] Lustre: srv-fs20676-OST0001: No data found on store. Initialize space. [10975.547668] Lustre: 309105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10975.617257] Lustre: fs20676-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10983.043613] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10984.934052] LustreError: 303662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.954952] LustreError: 303662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10991.925297] Lustre: server umount fs20676-OST0000 complete [11002.013969] Lustre: server umount fs20676-OST0001 complete [11014.457686] loop1: detected capacity change from 0 to 204800000 [11019.016278] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11019.246177] Lustre: fs20676-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11025.655725] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11032.361282] loop2: detected capacity change from 0 to 204800000 [11039.618038] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11048.799286] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11059.953279] Lustre: server umount fs20676-OST0000 complete [11071.730773] Lustre: server umount fs20676-OST0001 complete [11082.214727] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11087.329368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11087.331448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.344628] LustreError: Skipped 1 previous similar message [11087.348377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11087.366365] Lustre: Skipped 1 previous similar message [11087.377594] Lustre: Skipped 1 previous similar message [11092.480705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11092.487325] Lustre: Skipped 1 previous similar message [11097.580826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.582395] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11097.585090] Lustre: Skipped 1 previous similar message [11099.104286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11099.301477] Lustre: server umount lustre-OST0000 complete [11104.226099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.253385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11107.813917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.710749] Lustre: server umount lustre-MDT0000 complete [11113.924427] LustreError: 301758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659333 with bad export cookie 1470525167295584872 [11113.946773] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.950427] LustreError: 301758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11114.405280] Lustre: server umount lustre-MDT0001 complete [11131.190795] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11135.433947] Key type lgssc unregistered [11135.774551] LNet: 315035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11135.785651] LNetError: 315035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11135.814177] LNet: Removed LNI 192.168.201.139@tcp [11137.223170] Key type .llcrypt unregistered [11137.225317] Key type ._llcrypt unregistered [11182.896968] Key type ._llcrypt registered [11182.898710] Key type .llcrypt registered [11183.046224] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [11199.093210] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11200.037406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11200.059824] alg: No test for adler32 (adler32-zlib) [11201.227044] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11201.515581] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11203.320220] Key type lgssc registered [11204.759594] Lustre: Echo OBD driver; http://www.lustre.org/ [11240.850470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11240.874102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.187209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11242.297663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11242.401777] Lustre: lustre-MDT0000: new disk, initializing [11242.516071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11242.531048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11248.570451] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11258.549049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11258.658798] Lustre: 318898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11258.719482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11258.724806] Lustre: Skipped 1 previous similar message [11258.779411] Lustre: lustre-MDT0001: new disk, initializing [11258.860671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11258.901172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11258.913771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11262.890645] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11271.881780] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11276.991599] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11285.314804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.667507] Lustre: lustre-OST0000: new disk, initializing [11285.673127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11285.680692] Lustre: 320865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11285.809955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11291.232684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11291.244178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11291.334590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11292.802980] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11302.051987] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11304.553730] Lustre: Mounted lustre-client - version 2.17.57_1_g5840bc4 [11306.173528] Lustre: Unmounted lustre-client [11319.781721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11319.788608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11319.800566] Lustre: Skipped 1 previous similar message [11319.821221] Lustre: Skipped 1 previous similar message [11324.595842] Lustre: server umount lustre-OST0000 complete [11330.023746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11330.032137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11330.035765] Lustre: Skipped 1 previous similar message [11335.139511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11335.149987] Lustre: Skipped 1 previous similar message [11335.412372] Lustre: server umount lustre-MDT0000 complete [11339.459896] LustreError: 318889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659559 with bad export cookie 17811275271575531943 [11339.470763] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11339.484234] LustreError: 318889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11339.912481] Lustre: server umount lustre-MDT0001 complete [11351.494244] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11354.954630] Key type lgssc unregistered [11357.216171] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [11357.408482] LNet: 323028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11357.424706] LNetError: 323028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11433.269093] LNet: Removed LNI 192.168.201.139@tcp [11434.150277] Key type .llcrypt unregistered [11434.152095] Key type ._llcrypt unregistered [11456.279910] Key type ._llcrypt registered [11456.282273] Key type .llcrypt registered [11456.413953] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [11474.595712] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11476.679689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11476.798934] alg: No test for adler32 (adler32-zlib) [11477.950191] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11478.204238] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11479.945300] Key type lgssc registered [11481.459195] Lustre: Echo OBD driver; http://www.lustre.org/ [11514.699267] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:22:13 (1786659733) [11522.114214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11522.149581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11523.460189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11523.527102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11523.669914] Lustre: lustre-MDT0000: new disk, initializing [11523.795976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11523.841913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11528.784494] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11540.091782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.247487] Lustre: 326252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11540.321864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11540.329564] Lustre: Skipped 1 previous similar message [11540.458570] Lustre: lustre-MDT0001: new disk, initializing [11540.617792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11540.657385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11540.672979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11545.894358] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11556.074388] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11563.087297] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11571.700855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.272241] Lustre: lustre-OST0000: new disk, initializing [11572.287568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11572.300198] Lustre: 328220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11572.431940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11581.031483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11581.046223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11581.245362] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11581.255523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11595.678664] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11612.136171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11612.148393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11612.148864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11612.164299] Lustre: Skipped 1 previous similar message [11617.258211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11617.274884] Lustre: Skipped 1 previous similar message [11617.991118] Lustre: server umount lustre-OST0000 complete [11627.496388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11627.496856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11627.525520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11627.541857] Lustre: Skipped 1 previous similar message [11627.576047] Lustre: Skipped 2 previous similar messages [11632.610609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.216197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11637.725692] Lustre: server umount lustre-MDT0000 complete [11642.851801] LustreError: 327415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.885670] LustreError: 327415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11643.104231] LustreError: 326245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659863 with bad export cookie 11692704674027909858 [11643.130061] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11643.135092] LustreError: 326245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11643.537578] Lustre: server umount lustre-MDT0001 complete [11659.764087] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11662.869641] Key type lgssc unregistered [11663.287646] LNet: 330136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11663.297205] LNetError: 330136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11663.321088] LNet: Removed LNI 192.168.201.139@tcp [11664.522281] Key type .llcrypt unregistered [11664.526299] Key type ._llcrypt unregistered [11686.028293] Key type ._llcrypt registered [11686.031905] Key type .llcrypt registered [11686.164753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11687.240298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11687.247667] alg: No test for adler32 (adler32-zlib) [11688.447514] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11688.713807] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11690.440420] Key type lgssc registered [11691.640305] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.819978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.861344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.328477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11704.439799] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11710.951654] LustreError: 331411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.983076] LustreError: 331411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11713.285651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.787177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11718.297468] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11725.574612] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11731.505121] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11740.718040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11741.126709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11747.501868] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11755.287845] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11766.836329] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11768.310183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11772.384649] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [11773.929863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11773.940768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11773.941809] Lustre: Skipped 2 previous similar messages [11773.953507] Lustre: Skipped 2 previous similar messages [11777.489142] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [11778.035029] Lustre: server umount lustre-MDT0000 complete [11779.053885] LustreError: 331408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11779.079092] LustreError: 331408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11781.343654] LustreError: 331391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660001 with bad export cookie 14443345456653812486 [11781.351716] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11781.360971] LustreError: 331391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11782.614827] LustreError: 331407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.657109] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [11784.161938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.209482] LustreError: 331407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.231221] LustreError: 331407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11787.811802] Lustre: server umount lustre-MDT0001 complete [11792.426647] Lustre: server umount lustre-OST0000 complete [11801.045738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11806.035114] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11808.990381] Lustre: 335591:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.139@tcp [11811.825672] Lustre: server umount MGS complete [11821.846570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.182404] LustreError: 336163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.236710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11827.480391] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11832.821256] LustreError: 336163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11832.857398] LustreError: 336163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11838.338444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.878794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11844.513584] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11854.582927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.075889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11863.148103] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11871.880500] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11873.604899] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11879.410655] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11880.665519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11887.082890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11887.089442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.105460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11887.108468] Lustre: Skipped 2 previous similar messages [11887.519450] Lustre: server umount lustre-OST0000 complete [11893.219973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11893.255560] Lustre: Skipped 1 previous similar message [11897.986368] Lustre: server umount lustre-MDT0000 complete [11898.338309] LustreError: 336892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11898.355749] LustreError: 336892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11902.723838] LustreError: 336143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660122 with bad export cookie 14443345456653813326 [11902.766024] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11912.328711] Lustre: server umount lustre-MDT0001 complete [11924.282993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11927.378937] Key type lgssc unregistered [11927.842208] LNet: 339603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11927.853995] LNetError: 339603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11927.896759] LNet: Removed LNI 192.168.201.139@tcp [11928.854960] Key type .llcrypt unregistered [11928.859738] Key type ._llcrypt unregistered [11948.496498] Key type ._llcrypt registered [11948.502024] Key type .llcrypt registered [11948.633665] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [11965.141317] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11966.465190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11966.586844] alg: No test for adler32 (adler32-zlib) [11967.819527] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11968.092272] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11969.851349] Key type lgssc registered [11971.084395] Lustre: Echo OBD driver; http://www.lustre.org/ [12003.138808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12003.165328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.537587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12004.571079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12004.683261] Lustre: lustre-MDT0000: new disk, initializing [12004.770387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12004.809734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12009.047907] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12020.642867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.780692] Lustre: 342768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12020.876425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12020.876841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12020.886154] Lustre: Skipped 1 previous similar message [12020.990575] Lustre: lustre-MDT0001: new disk, initializing [12021.135608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12021.188928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12021.197725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12025.249475] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12034.055957] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12040.576955] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12046.419653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12046.749993] Lustre: lustre-OST0000: new disk, initializing [12046.759664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12046.772952] Lustre: 344734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12046.890180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12052.980599] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12055.605439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12055.626654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12055.762797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12063.182655] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12068.504831] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12068.786721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12074.379662] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12074.576084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12081.653557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12081.659951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12081.669993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12081.681413] Lustre: Skipped 1 previous similar message [12081.698994] LustreError: Skipped 1 previous similar message [12083.694942] Lustre: server umount lustre-OST0000 complete [12087.777530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12087.784780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.805625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12087.808382] Lustre: Skipped 1 previous similar message [12091.874247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12091.882664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12093.560452] Lustre: server umount lustre-MDT0000 complete [12097.349954] LustreError: 342759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660317 with bad export cookie 4447729833840449657 [12097.361507] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12097.364605] LustreError: 342759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12097.652936] Lustre: server umount lustre-MDT0001 complete [12105.133246] Lustre: DEBUG MARKER: == conf-sanity test 163: pattern support negotiation visible in mdc and osc import ========================================================== 18:32:04 (1786660324) [12120.298684] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12128.946367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12129.564621] LustreError: 347865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.583237] LustreError: 347865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12129.649777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12134.557898] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12134.883160] LustreError: 347866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12138.977647] LustreError: 347865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12144.099020] LustreError: 347866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12144.635590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12145.127647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12150.874144] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12154.138436] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12164.496850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12164.933428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12172.389964] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12181.319774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12181.525729] Lustre: lustre-OST0001: new disk, initializing [12181.541498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12181.552185] Lustre: 350173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12181.662512] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12188.722388] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12191.348512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12191.359284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12191.486778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12199.605965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12209.261457] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12221.927558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12221.936569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.963564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12221.974509] Lustre: Skipped 1 previous similar message [12229.845617] Lustre: server umount lustre-MDT0000 complete [12232.674402] LustreError: 347866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12232.719301] LustreError: 347866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12233.582547] LustreError: 347846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660453 with bad export cookie 4447729833840450532 [12233.595723] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12233.599625] LustreError: 347846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12234.109049] Lustre: server umount lustre-MDT0001 complete [12248.527792] Lustre: server umount lustre-OST0000 complete [12262.847947] Lustre: server umount lustre-OST0001 complete [12266.858280] Lustre: DEBUG MARKER: == conf-sanity test 163a: mdt.*.pattern_types tunes advertised layout patterns ========================================================== 18:34:45 (1786660485) [12282.742642] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12292.569221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12293.037677] LustreError: 353154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.148215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.754721] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12306.439450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12311.973110] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12314.611475] Lustre: 354295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12321.227077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12327.941592] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12331.968897] LustreError: 354648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.011875] LustreError: 354648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12335.917383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12342.423342] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12350.401171] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12354.930560] Lustre: 356166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12367.183459] LNet: 356653:0:(debug.c:376:cfs_str2mask()) unknown mask 'bogus'. [12367.183459] mask usage: [+|-] ... [12383.714041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12383.718775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12383.724683] Lustre: Skipped 3 previous similar messages [12383.727037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12383.733698] Lustre: Skipped 8 previous similar messages [12386.701584] Lustre: server umount lustre-MDT0000 complete [12389.349241] LustreError: 353154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.373368] LustreError: 353154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12390.776604] LustreError: 355477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660610 with bad export cookie 4447729833840451988 [12390.784877] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12390.796195] LustreError: 355477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12391.143953] Lustre: server umount lustre-MDT0001 complete [12405.432771] Lustre: server umount lustre-OST0000 complete [12419.719696] Lustre: server umount lustre-OST0001 complete [12424.307879] Lustre: DEBUG MARKER: == conf-sanity test 163b: EC components are ignored when MDS does not negotiate parity ========================================================== 18:37:22 (1786660642) [12426.222086] Lustre: DEBUG MARKER: SKIP: conf-sanity test_163b needs >= 3 OSTs for EC [12429.471493] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:37:27 (1786660647) [12446.312367] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [12456.703030] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12490.595083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.844747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12490.884322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12490.963722] Lustre: lustre-MDT0000: new disk, initializing [12491.036310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12491.039093] Lustre: Skipped 3 previous similar messages [12491.056276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12494.857098] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12505.880727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12506.002978] Lustre: 360568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12506.058333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12506.061399] Lustre: Skipped 1 previous similar message [12506.123586] Lustre: lustre-MDT0001: new disk, initializing [12506.305745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12506.316146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12511.575141] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12520.223229] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12524.751707] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12530.790789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12531.049625] Lustre: lustre-OST0000: new disk, initializing [12531.053820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12531.060065] Lustre: 362534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12533.166318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12533.189416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12533.353300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12538.177680] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12546.456583] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12579.297498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12579.305696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12579.311856] LustreError: Skipped 1 previous similar message [12579.314216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12579.331817] Lustre: Skipped 4 previous similar messages [12579.359528] Lustre: Skipped 5 previous similar messages [12583.443580] Lustre: server umount lustre-OST0000 complete [12588.513329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12593.549751] Lustre: server umount lustre-MDT0000 complete [12594.669221] LustreError: 362543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12594.689841] LustreError: 362543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12597.217017] LustreError: 360558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660817 with bad export cookie 4447729833840453248 [12597.232117] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12597.234763] LustreError: 360558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12597.721268] Lustre: server umount lustre-MDT0001 complete [12611.972678] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12614.957270] Key type lgssc unregistered [12615.254960] LNet: 364710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12615.265499] LNetError: 364710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12615.287219] LNet: Removed LNI 192.168.201.139@tcp [12616.181475] Key type .llcrypt unregistered [12616.183534] Key type ._llcrypt unregistered [12700.598807] Key type ._llcrypt registered [12700.600635] Key type .llcrypt registered [12700.691636] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12700.941441] Key type .llcrypt unregistered [12700.943562] Key type ._llcrypt unregistered [12762.764583] Key type ._llcrypt registered [12762.770136] Key type .llcrypt registered [12762.894791] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12763.220808] Key type .llcrypt unregistered [12763.226711] Key type ._llcrypt unregistered [12788.437453] Key type ._llcrypt registered [12788.439460] Key type .llcrypt registered [12788.546887] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12788.859367] Key type .llcrypt unregistered [12788.865220] Key type ._llcrypt unregistered [12821.228409] Key type ._llcrypt registered [12821.231735] Key type .llcrypt registered [12821.363564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12821.634455] Key type .llcrypt unregistered [12821.640914] Key type ._llcrypt unregistered [12898.569788] Key type ._llcrypt registered [12898.572279] Key type .llcrypt registered [12898.676976] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12898.947853] Key type .llcrypt unregistered [12898.949816] Key type ._llcrypt unregistered [12929.750126] Key type ._llcrypt registered [12929.753701] Key type .llcrypt registered [12929.905310] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12930.308537] Key type .llcrypt unregistered [12930.312213] Key type ._llcrypt unregistered [12960.826616] Key type ._llcrypt registered [12960.829199] Key type .llcrypt registered [12960.915816] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12961.205938] Key type .llcrypt unregistered [12961.209411] Key type ._llcrypt unregistered [12988.080200] Key type ._llcrypt registered [12988.082195] Key type .llcrypt registered [12988.242590] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12988.523434] Key type .llcrypt unregistered [12988.527460] Key type ._llcrypt unregistered [13017.904135] Key type ._llcrypt registered [13017.906239] Key type .llcrypt registered [13018.070018] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13018.559405] Key type .llcrypt unregistered [13018.561635] Key type ._llcrypt unregistered [13052.658559] Key type ._llcrypt registered [13052.661966] Key type .llcrypt registered [13052.759457] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13053.234858] Key type .llcrypt unregistered [13053.236414] Key type ._llcrypt unregistered [13121.513231] Key type ._llcrypt registered [13121.515489] Key type .llcrypt registered [13121.628165] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13121.886516] Key type .llcrypt unregistered [13121.888782] Key type ._llcrypt unregistered [13152.056798] Key type ._llcrypt registered [13152.059542] Key type .llcrypt registered [13152.193748] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13152.550629] Key type .llcrypt unregistered [13152.552770] Key type ._llcrypt unregistered [13181.996757] Key type ._llcrypt registered [13181.998871] Key type .llcrypt registered [13182.109913] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13182.414578] Key type .llcrypt unregistered [13182.417654] Key type ._llcrypt unregistered [13237.751210] Key type ._llcrypt registered [13237.756156] Key type .llcrypt registered [13237.849546] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13252.486704] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13253.271404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13253.465804] alg: No test for adler32 (adler32-zlib) [13254.427533] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [13254.675477] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13256.417088] Key type lgssc registered [13257.387193] Lustre: Echo OBD driver; http://www.lustre.org/ [13296.201722] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13296.231974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13297.711797] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13297.774834] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13297.885516] Lustre: tst250fs-MDT0000: new disk, initializing [13298.009567] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13298.049624] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13304.266710] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13320.617058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13320.857243] Lustre: 384974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13320.985932] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13320.993901] Lustre: Skipped 1 previous similar message [13321.132320] Lustre: tst250fs-MDT0001: new disk, initializing [13321.308432] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13321.423213] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13321.484562] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13327.115720] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13332.794189] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13344.503212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13344.884251] Lustre: tst250fs-OST0000: new disk, initializing [13344.889569] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13344.895632] Lustre: 386610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13344.984822] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13350.483207] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13350.504499] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13350.634907] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13352.972270] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13363.668310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13363.826685] Lustre: tst250fs-OST0001: new disk, initializing [13363.831645] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13363.838047] Lustre: 387480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13363.931937] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13371.048923] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13371.476525] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13371.485374] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13371.581659] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13377.582622] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13380.223621] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13380.229893] Lustre: Skipped 1 previous similar message [13402.600953] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13402.603442] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13402.637639] Lustre: Skipped 2 previous similar messages [13402.654103] Lustre: Skipped 3 previous similar messages [13405.766315] Lustre: server umount tst250fs-MDT0000 complete [13410.263214] LustreError: 384968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661630 with bad export cookie 12832943093484631153 [13410.269668] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13410.299960] LustreError: 384968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13410.936921] Lustre: server umount tst250fs-MDT0001 complete [13425.328923] Lustre: server umount tst250fs-OST0000 complete [13428.705093] Lustre: 383200:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786661632/real 1786661632] req@ffff8f3c810bf100 x1873450341834112/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786661648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13428.731618] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13428.740204] Lustre: Skipped 1 previous similar message [13429.960516] Lustre: server umount tst250fs-OST0001 complete [13452.439217] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13461.533966] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13496.311755] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:55:14 (1786661714) [13498.293727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13516.145456] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13530.139702] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13571.496840] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13290 sec ========= 18:56:30 (1786661790) [13573.586277] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:56:32 (1786661792) === [13577.988532] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:56:36 (1786661796) === [13624.765306] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13627.727645] Key type lgssc unregistered [13628.148285] LNet: 396626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13628.156091] LNetError: 396626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13628.175809] LNet: Removed LNI 192.168.201.139@tcp [13629.306193] Key type .llcrypt unregistered [13629.307869] Key type ._llcrypt unregistered