[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1551791125 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003097] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.021675] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028251] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029383] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032029] *** VALIDATE tmpfs *** [ 0.034378] *** VALIDATE proc *** [ 0.035296] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042014] Speculative Store Bypass: Vulnerable [ 0.045000] debug: unmapping init [mem 0xffffffff8c859000-0xffffffff8c860fff] [ 0.047305] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048956] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050027] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055013] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057286] rcu: Hierarchical SRCU implementation. [ 0.059954] smp: Bringing up secondary CPUs ... [ 0.060841] x86: Booting SMP configuration: [ 0.061048] .... node #0, CPUs: #1 #2 #3 [ 0.108551] smp: Brought up 1 node, 4 CPUs [ 0.110012] smpboot: Max logical packages: 1 [ 0.111013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151259] node 0 deferred pages initialised in 38ms [ 0.158446] devtmpfs: initialized [ 0.159375] x86/mm: Memory block size: 128MB [ 0.163215] gcov: version magic: 0x41383552 [ 0.166776] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168382] pinctrl core: initialized pinctrl subsystem [ 0.169227] [ 0.169929] ************************************************************* [ 0.170018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171015] ** ** [ 0.172016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173124] ** ** [ 0.174017] ** This means that this kernel is built to expose internal ** [ 0.175013] ** IOMMU data structures, which may compromise security on ** [ 0.176020] ** your system. ** [ 0.177018] ** ** [ 0.178020] ** If you see this message and you are not debugging the ** [ 0.179015] ** kernel, report this immediately to your vendor! ** [ 0.180019] ** ** [ 0.181016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182013] ************************************************************* [ 0.184221] NET: Registered protocol family 16 [ 0.186502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190000] cpuidle: using governor menu [ 0.192000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194652] PCI: Using configuration type 1 for base access [ 0.197140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212295] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218105] cryptd: max_cpu_qlen set to 1000 [ 0.220951] ACPI: Added _OSI(Module Device) [ 0.221000] ACPI: Added _OSI(Processor Device) [ 0.223018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234729] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244724] ACPI: Interpreter enabled [ 0.246076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248020] ACPI: Using IOAPIC for interrupt routing [ 0.251147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255781] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268907] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276201] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280893] acpiphp: Slot [2] registered [ 0.283128] acpiphp: Slot [5] registered [ 0.285111] acpiphp: Slot [6] registered [ 0.287147] acpiphp: Slot [7] registered [ 0.288115] acpiphp: Slot [8] registered [ 0.290207] acpiphp: Slot [9] registered [ 0.291195] acpiphp: Slot [10] registered [ 0.292195] acpiphp: Slot [3] registered [ 0.294146] acpiphp: Slot [4] registered [ 0.295240] acpiphp: Slot [11] registered [ 0.297090] acpiphp: Slot [12] registered [ 0.299107] acpiphp: Slot [13] registered [ 0.300086] acpiphp: Slot [14] registered [ 0.301178] acpiphp: Slot [15] registered [ 0.302093] acpiphp: Slot [16] registered [ 0.304089] acpiphp: Slot [17] registered [ 0.305110] acpiphp: Slot [18] registered [ 0.307283] acpiphp: Slot [19] registered [ 0.309100] acpiphp: Slot [20] registered [ 0.310097] acpiphp: Slot [21] registered [ 0.312107] acpiphp: Slot [22] registered [ 0.314141] acpiphp: Slot [23] registered [ 0.316146] acpiphp: Slot [24] registered [ 0.318258] acpiphp: Slot [25] registered [ 0.319220] acpiphp: Slot [26] registered [ 0.321148] acpiphp: Slot [27] registered [ 0.323100] acpiphp: Slot [28] registered [ 0.325109] acpiphp: Slot [29] registered [ 0.326303] acpiphp: Slot [30] registered [ 0.327160] acpiphp: Slot [31] registered [ 0.329072] PCI host bridge to bus 0000:00 [ 0.331022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340253] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347404] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351142] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365866] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381749] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384685] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422156] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.475057] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519293] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.546022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.577875] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.591016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.618140] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.627024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660945] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.673018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.682020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.691917] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.694568] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696525] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706116] iommu: Default domain type: Passthrough [ 0.707000] SCSI subsystem initialized [ 0.708254] ACPI: bus type USB registered [ 0.710205] usbcore: registered new interface driver usbfs [ 0.712195] usbcore: registered new interface driver hub [ 0.714301] usbcore: registered new device driver usb [ 0.716308] pps_core: LinuxPPS API ver. 1 registered [ 0.718012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721090] PTP clock support registered [ 0.723065] EDAC MC: Ver: 3.0.0 [ 0.724169] PCI: Using ACPI for IRQ routing [ 0.725000] NetLabel: Initializing [ 0.726016] NetLabel: domain hash size = 128 [ 0.728025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729254] NetLabel: unlabeled traffic allowed by default [ 0.731509] vgaarb: loaded [ 0.733696] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.740415] clocksource: Switched to clocksource kvm-clock [ 0.911881] VFS: Disk quotas dquot_6.6.0 [ 0.913197] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.915111] *** VALIDATE ramfs *** [ 0.916171] *** VALIDATE hugetlbfs *** [ 0.917619] pnp: PnP ACPI init [ 0.919884] pnp: PnP ACPI: found 6 devices [ 0.935238] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.938129] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.939714] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.941378] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.943429] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945436] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.947697] NET: Registered protocol family 2 [ 0.949904] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.954770] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957554] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.962269] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965388] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967922] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.970399] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972635] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975084] NET: Registered protocol family 1 [ 0.977220] RPC: Registered named UNIX socket transport module. [ 0.978941] RPC: Registered udp transport module. [ 0.980345] RPC: Registered tcp transport module. [ 0.981631] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983509] NET: Registered protocol family 44 [ 0.984773] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.986344] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987860] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989603] PCI: CLS 0 bytes, default 64 [ 0.990729] Unpacking initramfs... [ 2.906479] debug: unmapping init [mem 0xffff8a553cc54000-0xffff8a553ffbffff] [ 2.909705] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.911239] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.913171] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.543488] Initialise system trusted keyrings [ 3.545234] Key type blacklist registered [ 3.547381] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.557540] zbud: loaded [ 3.561087] *** VALIDATE nfs *** [ 3.562580] *** VALIDATE nfs4 *** [ 3.564187] pstore: using deflate compression [ 3.568358] Platform Keyring initialized [ 3.687898] NET: Registered protocol family 38 [ 3.689980] Key type asymmetric registered [ 3.691620] Asymmetric key parser 'x509' registered [ 3.693702] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.697906] io scheduler mq-deadline registered [ 3.699668] io scheduler kyber registered [ 3.701616] io scheduler bfq registered [ 3.703489] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.706810] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.710529] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.713919] ACPI: Power Button [PWRF] [ 3.725313] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.737335] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.783986] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.799964] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.839192] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.867686] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.902579] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.908706] Non-volatile memory driver v1.3 [ 3.910481] Linux agpgart interface v0.103 [ 3.943459] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.946519] vda: detected capacity change from 0 to 74711040 [ 3.969918] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.973049] vdb: detected capacity change from 0 to 1073741824 [ 3.992270] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.995105] vdc: detected capacity change from 0 to 2621440000 [ 4.046819] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.052517] vdd: detected capacity change from 0 to 2621440000 [ 4.080596] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.083769] vde: detected capacity change from 0 to 4294967296 [ 4.103819] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.107315] vdf: detected capacity change from 0 to 4294967296 [ 4.118747] libphy: Fixed MDIO Bus: probed [ 4.172116] usbcore: registered new interface driver usbserial_generic [ 4.179212] usbserial: USB Serial support registered for generic [ 4.184302] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.191848] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.195367] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.200217] mousedev: PS/2 mouse device common for all mice [ 4.205837] rtc_cmos 00:05: RTC can wake from S4 [ 4.212563] rtc_cmos 00:05: registered as rtc0 [ 4.216798] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.221453] intel_pstate: CPU model not supported [ 4.226554] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.228209] hid: raw HID events driver (C) Jiri Kosina [ 4.240132] usbcore: registered new interface driver usbhid [ 4.248723] usbhid: USB HID core driver [ 4.251173] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.251381] drop_monitor: Initializing network drop monitor service [ 4.264332] Initializing XFRM netlink socket [ 4.268804] NET: Registered protocol family 10 [ 4.273083] Segment Routing with IPv6 [ 4.273730] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.279601] NET: Registered protocol family 17 [ 4.279956] mpls_gso: MPLS GSO support [ 4.300501] RAS: Correctable Errors collector initialized. [ 4.303824] AVX version of gcm_enc/dec engaged. [ 4.306825] AES CTR mode by8 optimization enabled [ 4.685736] sched_clock: Marking stable (4685648135, 0)->(6108102371, -1422454236) [ 4.689854] registered taskstats version 1 [ 4.695339] Loading compiled-in X.509 certificates [ 4.701158] zswap: loaded using pool lzo/zbud [ 4.773223] Key type big_key registered [ 4.796082] Key type encrypted registered [ 4.801539] ima: No TPM chip found, activating TPM-bypass! [ 4.804529] ima: Allocated hash algorithm: sha1 [ 4.806938] ima: No architecture policies found [ 4.809597] evm: Initialising EVM extended attributes: [ 4.812857] evm: security.selinux [ 4.814652] evm: security.ima [ 4.816208] evm: security.capability [ 4.817993] evm: HMAC attrs: 0x1 [ 4.821628] rtc_cmos 00:05: setting system clock to 2026-08-17 06:09:35 UTC (1786946975) [ 4.831608] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 4.836204] debug: unmapping init [mem 0xffffffff8c582000-0xffffffff8c858fff] [ 4.844263] Write protecting the kernel read-only data: 28672k [ 4.848761] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 4.852143] debug: unmapping init [mem 0xffffffff8b514000-0xffffffff8b5fffff] [ 5.117477] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.135178] systemd[1]: Detected virtualization kvm. [ 5.139551] systemd[1]: Detected architecture x86-64. [ 5.142650] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.256145] systemd[1]: No hostname configured. [ 5.257828] systemd[1]: Set hostname to . [ 5.260453] random: systemd: uninitialized urandom read (16 bytes read) [ 5.263140] systemd[1]: Initializing machine ID from random generator. [ 5.689911] random: ln: uninitialized urandom read (6 bytes read) [ 6.193488] random: systemd: uninitialized urandom read (16 bytes read) [ 6.202769] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.214881] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.219328] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.887615] device-mapper: uevent: version 1.0.3 [ 7.890622] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 9.596821] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached targe[ 10.297805] random: fast init done t Basic System. [ OK [0[ 10.713381] scsi host0: ata_piix m] Started Hardware RNG Entropy Gatherer Daemon. [ 11.077938] scsi host1: ata_piix [ 11.636830] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.641091] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.400478] random: crng init done [ 18.434460] random: 7 urandom warning(s) missed due to ratelimiting [ 21.691651] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.286985] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.918545] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.880695] SELinux: Disabled at runtime. [ 29.113139] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.149504] systemd[1]: Detected virtualization kvm. [ 29.151115] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 32.623553] systemd[1]: initrd-switch-root.service: Succeeded. [ 32.649281] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 32.673083] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 32.695059] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 32.702721] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 32.740334] systemd[1]: Starting Journal Service... Starting Journal Service... [ 32.764818] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ 33.639120] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 35.359138] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.254579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.533249] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.835441] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.085643] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 47.401609] hrtimer: interrupt took 3858279 ns [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 49.083905] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit)[ 50.498172] NFS: Registering the id_resolver key type [ 50.508494] Key type id_resolver registered [ 50.514237] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg130-server login: [ 109.700692] libcfs: loading out-of-tree module taints kernel. [ 109.772573] Key type ._llcrypt registered [ 109.778757] Key type .llcrypt registered [ 109.867316] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 128.269335] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 129.710937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 129.722469] alg: No test for adler32 (adler32-zlib) [ 131.178563] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 132.101445] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 133.879171] Key type lgssc registered [ 135.693520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 149.987493] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.540520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 207.234341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.258853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.445859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.479730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.569625] Lustre: lustre-MDT0000: new disk, initializing [ 208.622095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.635374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.717960] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.416976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.522083] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.573403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.581411] Lustre: Skipped 1 previous similar message [ 228.688284] Lustre: lustre-MDT0001: new disk, initializing [ 228.796868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.812854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.818371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.046023] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.857981] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.064283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.402769] Lustre: lustre-OST0000: new disk, initializing [ 248.409538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.416882] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.465462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.034937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.053345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.160160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 255.541702] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.611197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.848428] Lustre: lustre-OST0001: new disk, initializing [ 269.865331] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.870194] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.985618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.240915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.070891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.087235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.164280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.877338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.186663] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 307.655928] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.847844] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing yml_node [ 320.040320] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 323.100794] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 326.351664] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 328.232611] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 02:14:57 EDT 2026 [ 347.045844] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 348.509861] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 366.048243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.054231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.076993] Lustre: Skipped 3 previous similar messages [ 366.101104] Lustre: Skipped 3 previous similar messages [ 371.183177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.189021] Lustre: Skipped 3 previous similar messages [ 376.288890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.295994] Lustre: Skipped 3 previous similar messages [ 378.335148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.581620] Lustre: server umount lustre-MDT0000 complete [ 382.219152] LustreError: 7922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947352 with bad export cookie 4755266031589768033 [ 382.222545] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.229327] LustreError: 7922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 386.529922] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.535557] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.552832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.573281] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 386.617674] Lustre: Skipped 1 previous similar message [ 387.552703] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.577108] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 388.902355] Lustre: server umount lustre-MDT0001 complete [ 393.508256] Lustre: server umount lustre-OST0000 complete [ 397.330261] Lustre: server umount lustre-OST0001 complete [ 405.938148] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 420.821269] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 459.527795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.868111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.891670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.995609] Lustre: lustre-MDT0000: new disk, initializing [ 460.095948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.110161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 464.600666] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 475.792929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.917788] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 475.989659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 475.993333] Lustre: Skipped 1 previous similar message [ 476.116305] Lustre: lustre-MDT0001: new disk, initializing [ 476.269886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 476.335898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 476.343725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 480.934784] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 490.197756] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.672281] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.266304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.635459] Lustre: lustre-OST0000: new disk, initializing [ 504.641186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.653283] Lustre: 17055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.745441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.416407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 506.437188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 506.568657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 511.362441] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 521.380931] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 525.895952] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 526.086787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 530.502264] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 530.704904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 537.569770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.571785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.578042] LustreError: Skipped 1 previous similar message [ 537.583663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.587073] Lustre: Skipped 1 previous similar message [ 537.600613] Lustre: Skipped 4 previous similar messages [ 547.295199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.569825] Lustre: server umount lustre-OST0000 complete [ 552.931386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 552.933297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.942245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.942257] Lustre: Skipped 2 previous similar messages [ 552.971673] Lustre: Skipped 1 previous similar message [ 558.687070] Lustre: server umount lustre-MDT0000 complete [ 563.175094] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.191282] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 563.336288] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947534 with bad export cookie 4755266031589769566 [ 563.364580] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.638289] Lustre: server umount lustre-MDT0001 complete [ 569.610180] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:18:59 (1786947539) [ 571.806656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 573.552142] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:19:03 (1786947543) [ 580.755110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.624707] LustreError: 19467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.755397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.706419] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 587.246076] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.660063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.805929] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 610.378512] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.916171] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 624.184507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 631.396650] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 638.693734] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 651.237238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.247612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.249680] Lustre: Skipped 1 previous similar message [ 651.255919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.263847] Lustre: Skipped 4 previous similar messages [ 656.821045] Lustre: server umount lustre-OST0000 complete [ 661.475761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.508938] Lustre: Skipped 1 previous similar message [ 667.881588] Lustre: server umount lustre-MDT0000 complete [ 671.713918] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.741520] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 672.328513] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947643 with bad export cookie 4755266031589770448 [ 672.339443] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 672.366976] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.625461] Lustre: server umount lustre-MDT0001 complete [ 684.985982] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 688.475502] Key type lgssc unregistered [ 688.806873] LNet: 23003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.811124] LNetError: 23003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.823167] LNet: Removed LNI 192.168.201.130@tcp [ 689.955172] Key type .llcrypt unregistered [ 689.957376] Key type ._llcrypt unregistered [ 714.465621] Key type ._llcrypt registered [ 714.468496] Key type .llcrypt registered [ 714.591341] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 715.977755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.171982] alg: No test for adler32 (adler32-zlib) [ 717.328716] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 717.591245] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 719.360607] Key type lgssc registered [ 720.687533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 730.397662] loop: module loaded [ 730.505306] loop0: detected capacity change from 0 to 204800000 [ 730.780325] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.803219] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 731.062589] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.768193] loop0: detected capacity change from 0 to 204800000 [ 737.952655] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.965987] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.990851] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.491796] loop0: detected capacity change from 0 to 204800000 [ 750.066594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.087959] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.427675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 751.534676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 751.634592] Lustre: lustre-MDT0000: new disk, initializing [ 751.764101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.816890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 756.590833] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 767.663929] loop1: detected capacity change from 0 to 204800000 [ 774.265311] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.637212] Lustre: lustre-OST07c6: new disk, initializing [ 774.640796] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 774.644981] Lustre: Skipped 1 previous similar message [ 774.652562] Lustre: 26194:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 774.734439] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 781.979415] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 782.962646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 782.973174] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 783.021688] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 788.070067] Lustre: 25183:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 788.090949] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 788.096228] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 788.101190] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 788.105702] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 788.110492] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 803.250650] Lustre: 25182:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 803.256521] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.261678] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.266969] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 803.275501] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.285211] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.635645] Lustre: 25183:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 808.643408] Lustre: 25183:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.660724] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 808.678614] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.689017] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.697027] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.701373] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 808.710404] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.718104] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 808.725732] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.748748] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.761353] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.066792] Lustre: 25182:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 812.082482] Lustre: 25182:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 812.094501] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 812.107520] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.141132] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.156966] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.174084] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 812.196764] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.208885] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 812.215051] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.222641] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.232268] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.783740] Lustre: 25183:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 817.793225] Lustre: 25183:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 817.797988] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 817.808790] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.815503] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.821894] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.828307] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 817.834478] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.839151] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 817.844224] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.849332] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.853021] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 826.600878] Lustre: 26877:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 826.617418] Lustre: 26877:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 826.631145] Lustre: 26877:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 826.653196] Lustre: 26877:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.663937] Lustre: 26877:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 826.682257] Lustre: 26877:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.694500] Lustre: 26877:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 826.701922] Lustre: 26877:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.713958] Lustre: 26877:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 826.727591] Lustre: 26877:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.742442] Lustre: 26877:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 826.750615] Lustre: 26877:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 880.103229] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.109656] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.126614] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 885.217485] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 885.506753] Lustre: server umount lustre-OST07c6 complete [ 895.996489] Lustre: server umount lustre-MDT0000 complete [ 916.016487] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 919.081771] Key type lgssc unregistered [ 919.320429] LNet: 28756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.343204] LNetError: 28756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.367861] LNet: Removed LNI 192.168.201.130@tcp [ 920.314762] Key type .llcrypt unregistered [ 920.317932] Key type ._llcrypt unregistered [ 942.006736] Key type ._llcrypt registered [ 942.009478] Key type .llcrypt registered [ 942.125777] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 943.269221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.348757] alg: No test for adler32 (adler32-zlib) [ 944.447369] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 944.705103] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 946.367752] Key type lgssc registered [ 947.748341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 958.899318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 958.941223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.638594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.095469] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 971.237711] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.251735] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 976.357153] LustreError: 30029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.608532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.111183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.902512] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 992.903946] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.935701] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1007.683670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.315994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.259899] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1024.640813] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.861250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1050.595805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1050.612892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.626158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.640560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.643128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.643596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.655151] Lustre: Skipped 1 previous similar message [ 1055.144376] Lustre: server umount lustre-OST0000 complete [ 1061.859283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1061.867937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.886880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.448095] Lustre: server umount lustre-MDT0000 complete [ 1068.002878] LustreError: 30029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.804959] LustreError: 30009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948040 with bad export cookie 13417741441362746252 [ 1069.805845] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.827792] LustreError: 30009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1070.352229] Lustre: server umount lustre-MDT0001 complete [ 1083.781309] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1086.702609] Key type lgssc unregistered [ 1087.185333] LNet: 33792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.197396] LNetError: 33792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.214739] LNet: Removed LNI 192.168.201.130@tcp [ 1088.069447] Key type .llcrypt unregistered [ 1088.078044] Key type ._llcrypt unregistered [ 1110.485344] Key type ._llcrypt registered [ 1110.493141] Key type .llcrypt registered [ 1110.633801] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1111.703836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.725027] alg: No test for adler32 (adler32-zlib) [ 1112.955778] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 1113.251740] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1114.919379] Key type lgssc registered [ 1116.294646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.863813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.889337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.443837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.435567] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1136.615754] LustreError: 35066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.660587] LustreError: 35066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1141.450800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.763448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.327361] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1154.325394] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.268741] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1171.030706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.449523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.493398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1179.400521] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1188.723913] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1203.181712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.186592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.196383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.236796] Lustre: Skipped 1 previous similar message [ 1208.292389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.309317] Lustre: Skipped 2 previous similar messages [ 1213.432352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.447428] Lustre: Skipped 1 previous similar message [ 1216.995377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1217.167062] Lustre: server umount lustre-OST0000 complete [ 1223.658222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1223.658463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.662981] LustreError: Skipped 1 previous similar message [ 1223.671899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.674924] Lustre: Skipped 1 previous similar message [ 1223.698067] Lustre: Skipped 1 previous similar message [ 1227.894677] Lustre: server umount lustre-MDT0000 complete [ 1228.771579] LustreError: 37674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.819773] LustreError: 37674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1233.835412] LustreError: 35801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948204 with bad export cookie 3623187416796408396 [ 1233.851231] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.853727] LustreError: 35801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1234.227342] Lustre: server umount lustre-MDT0001 complete [ 1247.827566] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1252.026345] Key type lgssc unregistered [ 1252.490283] LNet: 38633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.503453] LNetError: 38633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.548079] LNet: Removed LNI 192.168.201.130@tcp [ 1254.097809] Key type .llcrypt unregistered [ 1254.101538] Key type ._llcrypt unregistered [ 1286.131680] Key type ._llcrypt registered [ 1286.133885] Key type .llcrypt registered [ 1286.355404] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1288.740545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1288.834142] alg: No test for adler32 (adler32-zlib) [ 1290.324898] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 1290.506964] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1292.175758] Key type lgssc registered [ 1294.173579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1303.777472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1303.789942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.230709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.989867] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1314.793481] LustreError: 40051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.804964] LustreError: 40051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1319.908413] LustreError: 40052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.942028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.188185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1325.122447] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1332.878927] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1338.434023] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1346.086427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1346.398382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.525271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1354.255539] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1362.681980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1369.704464] Lustre: server umount lustre-MDT0000 complete [ 1370.082789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.095530] LustreError: 40780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.112707] Lustre: Skipped 1 previous similar message [ 1370.169040] LustreError: 40780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1373.651087] LustreError: 40032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948344 with bad export cookie 4783849256591349920 [ 1373.659133] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.666684] LustreError: 40032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1375.202472] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.206823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.228943] LustreError: 40046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1375.246761] Lustre: Skipped 1 previous similar message [ 1375.257507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1378.208774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1378.212913] Lustre: Skipped 1 previous similar message [ 1379.975892] Lustre: server umount lustre-MDT0001 complete [ 1400.765556] Lustre: server umount lustre-OST0000 complete [ 1417.174981] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1419.893400] Key type lgssc unregistered [ 1420.152191] LNet: 43915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.161309] LNetError: 43915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.221455] LNet: Removed LNI 192.168.201.130@tcp [ 1421.900326] Key type .llcrypt unregistered [ 1421.902173] Key type ._llcrypt unregistered [ 1445.916934] Key type ._llcrypt registered [ 1445.918955] Key type .llcrypt registered [ 1446.001659] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1447.093052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.122985] alg: No test for adler32 (adler32-zlib) [ 1448.312333] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 1448.587360] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1450.296046] Key type lgssc registered [ 1451.506943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1458.793873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1458.816113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.473064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.946324] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1470.951978] LustreError: 45334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.966542] LustreError: 45334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1473.202295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.646394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.650182] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1484.132953] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1491.311825] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1499.795050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.304995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.560118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1509.809975] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1521.227430] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1528.074155] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1529.312086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.334257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.345548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.368657] Lustre: Skipped 1 previous similar message [ 1529.408366] Lustre: Skipped 1 previous similar message [ 1532.919557] Lustre: server umount lustre-OST0000 complete [ 1554.917033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.938162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.948214] Lustre: Skipped 1 previous similar message [ 1556.693605] Lustre: server umount lustre-MDT0000 complete [ 1560.039708] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.059916] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1560.300684] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948530 with bad export cookie 15238914233478533298 [ 1560.305512] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.309501] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1566.877050] Lustre: server umount lustre-MDT0001 complete [ 1579.583705] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1583.445752] Key type lgssc unregistered [ 1583.806725] LNet: 49044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.814773] LNetError: 49044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.847331] LNet: Removed LNI 192.168.201.130@tcp [ 1584.816534] Key type .llcrypt unregistered [ 1584.817852] Key type ._llcrypt unregistered [ 1608.585467] Key type ._llcrypt registered [ 1608.588152] Key type .llcrypt registered [ 1608.672368] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1610.208393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.247887] alg: No test for adler32 (adler32-zlib) [ 1611.426957] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 1611.768892] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1613.513636] Key type lgssc registered [ 1614.768330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1624.753847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1624.860839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.471840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.782152] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1636.834834] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.843601] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1641.364328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.631222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.143126] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1654.125891] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1660.607917] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1666.915657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1667.281357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.424639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1673.648711] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1680.505802] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.110810] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1693.248825] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1706.394465] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-17 2h37m54s [ 1708.073429] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1712.609866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.617878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.630598] Lustre: Skipped 1 previous similar message [ 1712.645952] Lustre: Skipped 2 previous similar messages [ 1713.697187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.988356] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1714.999549] Lustre: Skipped 3 previous similar messages [ 1716.992153] Lustre: server umount lustre-MDT0000 complete [ 1719.780688] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.815506] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1721.070674] LustreError: 52534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.494868] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1725.938858] LustreError: 53147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.949631] LustreError: 53147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1728.512305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.650567] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.906136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.976791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1729.984223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1733.784974] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1737.116740] Lustre: DEBUG MARKER: Wait for df (36729) ... [ 1738.539963] Lustre: DEBUG MARKER: done [ 1743.329080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.340801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.354040] Lustre: Skipped 1 previous similar message [ 1743.364235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.353542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.364559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.413476] Lustre: server umount lustre-OST0000 complete [ 1755.622952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.623221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.630477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.630489] Lustre: Skipped 3 previous similar messages [ 1755.647323] Lustre: Skipped 1 previous similar message [ 1758.459751] Lustre: server umount lustre-MDT0000 complete [ 1761.852536] LustreError: 50441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948732 with bad export cookie 14138574507455717935 [ 1761.868356] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1762.246555] Lustre: server umount lustre-MDT0001 complete [ 1772.005605] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1774.293332] Key type lgssc unregistered [ 1774.537652] LNet: 55386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.545248] LNetError: 55386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.569533] LNet: Removed LNI 192.168.201.130@tcp [ 1775.344109] Key type .llcrypt unregistered [ 1775.346492] Key type ._llcrypt unregistered [ 1815.450337] Key type ._llcrypt registered [ 1815.454332] Key type .llcrypt registered [ 1815.683601] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1817.477251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1817.826367] alg: No test for adler32 (adler32-zlib) [ 1819.149847] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 1819.545661] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1821.375249] Key type lgssc registered [ 1823.099363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.052415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1832.073304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.504354] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1833.553969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1833.756333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.387144] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1848.045830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.149391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1848.188402] Lustre: 57597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1848.488409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1852.543684] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1859.687384] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1865.953923] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1873.048350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.185681] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1873.327719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.563159] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1882.861888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1886.939458] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.856854] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-17 2h41m02s [ 1894.535422] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1902.574228] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1905.472423] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1905.534340] LustreError: 57604:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1925.886965] Lustre: lustre-MDT0000: Client 5b123ec4-9eb4-4a40-a061-e66b655c229c (at 192.168.201.30@tcp) reconnecting [ 1945.559152] LustreError: 57604:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1948.068524] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1950.712950] Lustre: DEBUG MARKER: done [ 1958.324717] Lustre: server umount lustre-OST0000 complete [ 1965.026205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.029450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.045303] Lustre: Skipped 1 previous similar message [ 1966.118211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.133112] Lustre: Skipped 2 previous similar messages [ 1972.198542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.210288] Lustre: Skipped 1 previous similar message [ 1972.468522] Lustre: server umount lustre-MDT0000 complete [ 1976.288331] Lustre: 56919:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786948930/real 1786948930] req@00000000efd91c62 x1873751614479616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786948946 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1976.292232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.319086] Lustre: 56919:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1976.639124] LustreError: 60356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948947 with bad export cookie 25940010242336601 [ 1976.645993] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.649886] LustreError: 60356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1976.974601] Lustre: server umount lustre-MDT0001 complete [ 1988.690485] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1991.906943] Key type lgssc unregistered [ 1992.234752] LNet: 61928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.246607] LNetError: 61928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.273859] LNet: Removed LNI 192.168.201.130@tcp [ 1993.196822] Key type .llcrypt unregistered [ 1993.200139] Key type ._llcrypt unregistered [ 2035.728562] Key type ._llcrypt registered [ 2035.730767] Key type .llcrypt registered [ 2035.814956] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2037.121380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.175266] alg: No test for adler32 (adler32-zlib) [ 2038.199280] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2038.400952] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2040.047215] Key type lgssc registered [ 2041.072691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2044.302302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2044.322664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.630720] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2045.664562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2045.914892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.990276] Lustre: Failing over lustre-MDT0000 [ 2047.333856] Lustre: server umount lustre-MDT0000 complete [ 2054.498925] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:43:44 (1786949024) [ 2061.276217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.825650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.220454] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2075.354604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.572254] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2075.650568] Lustre: 64706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2075.935425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2080.681143] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2088.084358] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2093.742712] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.091758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.335607] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2101.508158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.445394] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2110.671996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2115.473561] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2121.856807] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2131.432396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.443533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.456643] Lustre: Skipped 1 previous similar message [ 2131.472098] Lustre: Skipped 1 previous similar message [ 2135.820885] Lustre: server umount lustre-MDT0000 complete [ 2139.657449] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949110 with bad export cookie 10397297859264165273 [ 2139.660764] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.668371] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2139.941845] Lustre: server umount lustre-MDT0001 complete [ 2142.833405] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2146.950458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2156.950126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.047291] Lustre: 63627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949112/real 1786949112] req@00000000143ac7c7 x1873751844102656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2158.068891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.075511] Lustre: Skipped 1 previous similar message [ 2161.119121] Lustre: 63629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949115/real 1786949115] req@0000000035d616b7 x1873751844102784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.231253] Lustre: 63629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949117/real 1786949117] req@00000000ce9e8c41 x1873751844102912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.280839] LustreError: 63626:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce9e8c41 x1873751844103680/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.664771] LustreError: 66553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.690197] LustreError: 66553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2165.811969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.927794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2166.755476] Lustre: 63627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949120/real 1786949120] req@00000000fc0b1f73 x1873751844103168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2167.848641] LustreError: 66554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.867732] LustreError: 68475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.891718] LustreError: 68475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2170.566311] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2171.871463] Lustre: 63628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949125/real 1786949125] req@0000000012a4764a x1873751844103296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.924035] LustreError: 68477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.959104] LustreError: 68477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2178.972729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.051608] LustreError: 68476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.065651] LustreError: 68476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2179.249601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.819202] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2184.684727] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2186.802563] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2191.298770] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2197.314715] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2204.174426] Lustre: server umount lustre-MDT0000 complete [ 2205.153192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.163167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.180759] LustreError: 66554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.208521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.889379] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949178 with bad export cookie 10397297859264167555 [ 2207.902730] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.910437] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.312956] Lustre: server umount lustre-MDT0001 complete [ 2215.127048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2225.473445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.703363] Lustre: 63627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949183/real 1786949183] req@00000000877b6395 x1873751844129792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2228.722572] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.735031] Lustre: Skipped 1 previous similar message [ 2233.209782] LustreError: 66553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.230349] LustreError: 66553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2233.263861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.331638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2237.706349] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2246.574443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.056678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2252.083533] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2260.670522] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2266.242861] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2272.744480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.752912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.786465] LustreError: 69504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.801158] LustreError: 69504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2273.076875] Lustre: server umount lustre-MDT0000 complete [ 2276.803448] LustreError: 64698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949247 with bad export cookie 10397297859264168934 [ 2276.807269] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.814568] LustreError: 64698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2277.467599] Lustre: server umount lustre-MDT0001 complete [ 2284.610034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2289.201687] Lustre: DEBUG MARKER: files compared the same [ 2300.688529] Lustre: server umount lustre-OST0000 complete [ 2317.321114] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2319.941891] Key type lgssc unregistered [ 2320.352898] LNet: 74434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.367981] LNetError: 74434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.401893] LNet: Removed LNI 192.168.201.130@tcp [ 2321.324685] Key type .llcrypt unregistered [ 2321.326550] Key type ._llcrypt unregistered [ 2340.676668] Key type ._llcrypt registered [ 2340.678696] Key type .llcrypt registered [ 2340.756936] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2341.779068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2341.875347] alg: No test for adler32 (adler32-zlib) [ 2342.963182] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2343.240975] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2344.959193] Key type lgssc registered [ 2346.281752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.208164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.239926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.659599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.785895] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2366.955351] LustreError: 75833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.969345] LustreError: 75833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2368.889922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.230585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.323590] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2380.164409] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2384.840330] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2391.733374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.149140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.928579] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2401.396058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2405.560800] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.072701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.077792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.097875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.254954] Lustre: server umount lustre-OST0000 complete [ 2420.704573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2420.711833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.728528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.241953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.252419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.359768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.376320] Lustre: Skipped 2 previous similar messages [ 2428.453698] Lustre: server umount lustre-MDT0000 complete [ 2431.457530] LustreError: 75830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.479801] LustreError: 75830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2431.568527] LustreError: 75815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949402 with bad export cookie 6754762202333312102 [ 2431.584702] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.873741] Lustre: server umount lustre-MDT0001 complete [ 2442.708567] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2445.199718] Key type lgssc unregistered [ 2445.432984] LNet: 79365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.440564] LNetError: 79365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.461363] LNet: Removed LNI 192.168.201.130@tcp [ 2446.119951] Key type .llcrypt unregistered [ 2446.123484] Key type ._llcrypt unregistered [ 2467.361936] Key type ._llcrypt registered [ 2467.366862] Key type .llcrypt registered [ 2467.475626] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2468.346769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.449227] alg: No test for adler32 (adler32-zlib) [ 2469.463940] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2469.671147] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2471.359190] Key type lgssc registered [ 2472.326900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.958537] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.972898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.559789] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2499.679437] LustreError: 80736:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [ 2526.303275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.827700] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2536.754641] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.099418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.196779] Lustre: *** cfs_fail_loc=706, val=0*** [ 2543.466629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.519933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2546.862993] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2548.709142] LustreError: 80762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.725063] LustreError: 80762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2553.830758] LustreError: 80761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.070517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.433607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.016524] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2564.111992] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.151240] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2574.304578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.308388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.315095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.817346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.830587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.848503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.404113] Lustre: server umount lustre-OST0000 complete [ 2580.960834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.968600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.971114] Lustre: Skipped 1 previous similar message [ 2580.989369] Lustre: Skipped 3 previous similar messages [ 2586.080246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.087987] Lustre: Skipped 1 previous similar message [ 2586.776917] Lustre: server umount lustre-MDT0000 complete [ 2589.625978] LustreError: 80744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949560 with bad export cookie 6938253810014992191 [ 2589.636025] LustreError: 80744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2589.638062] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.917301] Lustre: server umount lustre-MDT0001 complete [ 2598.412363] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2600.568687] Key type lgssc unregistered [ 2600.807832] LNet: 84355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.813494] LNetError: 84355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.828310] LNet: Removed LNI 192.168.201.130@tcp [ 2601.435908] Key type .llcrypt unregistered [ 2601.438187] Key type ._llcrypt unregistered [ 2620.471470] Key type ._llcrypt registered [ 2620.474331] Key type .llcrypt registered [ 2620.584580] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2621.414572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.509980] alg: No test for adler32 (adler32-zlib) [ 2622.513795] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2622.788762] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2624.424623] Key type lgssc registered [ 2625.443356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2632.769469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2632.783279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.394230] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2643.745366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.267721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2647.721394] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2649.572186] LustreError: 86437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.579666] LustreError: 86437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2653.668912] LustreError: 86440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.690028] LustreError: 86440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2654.513896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.788513] LustreError: 86437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.812867] LustreError: 86437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2654.831355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.829232] LustreError: 86441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.841280] LustreError: 86441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2660.685736] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2664.929952] LustreError: 86436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.945386] LustreError: 86436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2667.996870] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.008399] Lustre: MGS: Client 11c9ef60-0c2d-46d3-9f5d-d63f26b5292f (at 0@lo) reconnecting [ 2668.026920] LustreError: 86451:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2668.155578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.207749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2671.466133] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2683.877307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.881898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.892274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.904858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.905820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.910526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.576309] Lustre: server umount lustre-OST0000 complete [ 2695.137990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.158205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.164219] Lustre: Skipped 2 previous similar messages [ 2697.754446] Lustre: server umount lustre-MDT0000 complete [ 2700.256366] LustreError: 86440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.279438] LustreError: 86440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2701.329581] Lustre: server umount lustre-MDT0001 complete [ 2704.178570] Lustre: server umount MGS complete [ 2715.418554] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:54:44 (1786949684) [ 2762.599594] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 2775.782802] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2814.028249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.529314] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2827.775651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.982905] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2828.028485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2828.121184] Lustre: lustre-MDT0001: new disk, initializing [ 2828.210911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2828.231177] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2828.239206] Lustre: 93607:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2828.247452] LustreError: 93607:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2832.333548] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2841.884873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2842.121352] Lustre: lustre-OST0000: new disk, initializing [ 2842.124542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2842.129765] Lustre: 94374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2842.175813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.453868] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2849.387184] Lustre: 94713:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000e14ff900 x1873752456468992/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786949836 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2856.839688] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2863.410217] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.418947] Lustre: MGS: Client ec3ef2c7-7cb6-4d96-90e4-9884e3110ad9 (at 0@lo) reconnecting [ 2863.456238] Lustre: 92878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2863.472770] LustreError: 93593:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2863.493477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2863.553546] Lustre: lustre-MDT0000: new disk, initializing [ 2863.666295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.697242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2863.719691] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2864.736197] Lustre: 95445:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b8001954 x1873752456475392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786949851 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2868.846994] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2869.737935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2869.757222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2869.965425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2894.819130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.823170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2894.835465] Lustre: Skipped 2 previous similar messages [ 2899.511541] Lustre: server umount lustre-OST0000 complete [ 2905.062943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.068222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2905.074320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2905.084704] Lustre: Skipped 1 previous similar message [ 2905.105575] Lustre: Skipped 2 previous similar messages [ 2909.966957] Lustre: server umount lustre-MDT0000 complete [ 2910.185380] LustreError: 93579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.206634] LustreError: 93579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2920.686299] Lustre: server umount lustre-MDT0001 complete [ 2924.526257] Lustre: server umount MGS complete [ 2940.057936] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:58:28 (1786949908) [ 2960.909095] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2964.325775] Key type lgssc unregistered [ 2964.639759] LNet: 97947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2964.645769] LNetError: 97947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2964.661311] LNet: Removed LNI 192.168.201.130@tcp [ 2965.345287] Key type .llcrypt unregistered [ 2965.347663] Key type ._llcrypt unregistered [ 2985.339583] Key type ._llcrypt registered [ 2985.341268] Key type .llcrypt registered [ 2985.482764] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2987.604601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2987.654776] alg: No test for adler32 (adler32-zlib) [ 2988.727462] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 2988.939441] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2990.591224] Key type lgssc registered [ 2991.923325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3000.265598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3000.293607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.008915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.715841] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3011.560367] LustreError: 99168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.591366] LustreError: 99168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3016.678516] LustreError: 99169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.229553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.592295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.626600] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3034.035394] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3040.538212] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3048.811272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.185716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.369260] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3063.883261] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3071.726680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3085.326581] Lustre: server umount lustre-OST0000 complete [ 3090.401994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3090.424911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.458059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.450563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.451972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.481404] Lustre: Skipped 1 previous similar message [ 3097.572324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.581294] Lustre: Skipped 1 previous similar message [ 3099.705676] Lustre: server umount lustre-MDT0000 complete [ 3100.639235] Lustre: 98479:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786950055/real 1786950055] req@00000000dbc35020 x1873752840239104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786950071 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3100.668235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.695164] LustreError: 99169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.718949] LustreError: 99169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3103.713350] Lustre: 98481:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786950058/real 1786950058] req@000000002e4726eb x1873752840239872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786950074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3104.107966] LustreError: 99150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950074 with bad export cookie 10655996069906887635 [ 3104.131063] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.456102] Lustre: server umount lustre-MDT0001 complete [ 3115.663600] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3119.558491] Key type lgssc unregistered [ 3120.113720] LNet: 102739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3120.120480] LNetError: 102739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3120.151970] LNet: Removed LNI 192.168.201.130@tcp [ 3121.337190] Key type .llcrypt unregistered [ 3121.349765] Key type ._llcrypt unregistered [ 3144.245499] Key type ._llcrypt registered [ 3144.249888] Key type .llcrypt registered [ 3144.453744] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3146.354355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.433497] alg: No test for adler32 (adler32-zlib) [ 3147.653403] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3147.959460] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3149.655540] Key type lgssc registered [ 3150.927605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.839991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3158.851119] LustreError: 103827:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3158.893711] LustreError: 103830:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3158.901679] LustreError: 103827:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3158.910823] LustreError: 103827:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3158.920611] LustreError: 103827:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3158.929503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.588968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.268816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.612830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.788233] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3181.752185] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing lsmod [ 3185.040161] LustreError: 105030:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3185.118870] LustreError: 105037:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3185.135335] LustreError: 105030:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4913 [ 3185.150531] LustreError: 105030:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3185.159321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.161594] LustreError: 105030:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3185.654436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.687838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3190.242289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.248473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.268260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.777497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.778757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.799475] Lustre: Skipped 1 previous similar message [ 3191.814269] Lustre: Skipped 2 previous similar messages [ 3194.922957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.394938] Lustre: server umount lustre-MDT0000 complete [ 3199.758240] LustreError: 103848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950170 with bad export cookie 14254582727069130798 [ 3199.774379] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3199.977219] LustreError: 103863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.985530] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.989765] LustreError: 103863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3200.010545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3200.013212] Lustre: Skipped 2 previous similar messages [ 3201.119397] LustreError: 104299:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ad3cffee x1873753006953728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3201.383811] Lustre: server umount lustre-MDT0001 complete [ 3205.935972] Lustre: server umount lustre-OST0000 complete [ 3217.168238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.835140] LustreError: 106100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.934079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.017300] LustreError: 106101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.195282] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3228.131834] LustreError: 106100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.250967] LustreError: 106101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.589377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.089642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.513631] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3249.927314] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.495873] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3265.413447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.202526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.467802] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3282.557661] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3287.753125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3293.633267] Lustre: server umount lustre-OST0000 complete [ 3298.275984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.293918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.312345] Lustre: Skipped 1 previous similar message [ 3298.331922] Lustre: Skipped 1 previous similar message [ 3307.778306] Lustre: server umount lustre-MDT0000 complete [ 3308.515164] LustreError: 106100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.535588] LustreError: 106100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3311.340381] Lustre: 103230:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786950265/real 1786950265] req@000000007438ff81 x1873753006983808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786950281 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3311.381197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.896842] LustreError: 106083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950283 with bad export cookie 14254582727069131722 [ 3312.918949] LustreError: 106083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3312.923431] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.633585] Lustre: server umount lustre-MDT0001 complete [ 3328.356329] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3332.192279] Key type lgssc unregistered [ 3332.512838] LNet: 109640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.518269] LNetError: 109640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3333.553988] LNet: Removed LNI 192.168.201.130@tcp [ 3334.826621] Key type .llcrypt unregistered [ 3334.829314] Key type ._llcrypt unregistered [ 3364.344414] Key type ._llcrypt registered [ 3364.346881] Key type .llcrypt registered [ 3364.459461] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3365.729471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3365.786245] alg: No test for adler32 (adler32-zlib) [ 3366.985684] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3367.350712] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3369.063225] Key type lgssc registered [ 3370.292587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3379.822286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3379.837706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.237386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3386.291089] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3390.443046] LustreError: 111058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.473805] LustreError: 111058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3395.553620] LustreError: 111059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.805669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.369673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.009616] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3414.198151] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3421.172041] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3430.225130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.667741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.715567] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3450.144235] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3451.182872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3456.699229] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3461.329813] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3471.846990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.852560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3471.863351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3471.870459] Lustre: Skipped 1 previous similar message [ 3474.820669] Lustre: server umount lustre-OST0000 complete [ 3482.086217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.129192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.140264] Lustre: Skipped 2 previous similar messages [ 3485.995473] Lustre: server umount lustre-MDT0000 complete [ 3487.206676] LustreError: 111059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.228694] LustreError: 111059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3490.299153] LustreError: 111038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950460 with bad export cookie 3463881851443375126 [ 3490.317589] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.324268] LustreError: 111038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3490.635238] Lustre: server umount lustre-MDT0001 complete [ 3504.286573] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3507.154991] Key type lgssc unregistered [ 3507.440450] LNet: 114704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3507.457753] LNetError: 114704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3507.472707] LNet: Removed LNI 192.168.201.130@tcp [ 3508.467808] Key type .llcrypt unregistered [ 3508.470164] Key type ._llcrypt unregistered [ 3529.094218] Key type ._llcrypt registered [ 3529.097526] Key type .llcrypt registered [ 3529.277543] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3531.643349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3531.766080] alg: No test for adler32 (adler32-zlib) [ 3533.049159] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3533.343690] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3535.007228] Key type lgssc registered [ 3536.516587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3545.198831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.217914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.713167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.787718] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3556.833420] LustreError: 115929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.856757] LustreError: 115929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3560.330405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.947206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3566.525247] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3575.274715] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3581.806890] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3589.628982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.103971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.210164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3597.989934] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3608.385409] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.839106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3620.849745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.866286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.868176] Lustre: Skipped 1 previous similar message [ 3622.147407] Lustre: server umount lustre-OST0000 complete [ 3627.493668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3627.516698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.534741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.544095] Lustre: Skipped 1 previous similar message [ 3631.086041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.087570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.198645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.207793] Lustre: Skipped 2 previous similar messages [ 3641.311659] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3641.318896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.366697] Lustre: Skipped 1 previous similar message [ 3641.668541] Lustre: server umount lustre-MDT0000 complete [ 3646.438953] LustreError: 115928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.469641] LustreError: 115928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3646.607119] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950617 with bad export cookie 17738744545018607237 [ 3646.615615] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.625646] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3647.289814] Lustre: server umount lustre-MDT0001 complete [ 3662.580453] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3666.152909] Key type lgssc unregistered [ 3666.538104] LNet: 119464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.555624] LNetError: 119464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.575496] LNet: Removed LNI 192.168.201.130@tcp [ 3667.746247] Key type .llcrypt unregistered [ 3667.748200] Key type ._llcrypt unregistered [ 3701.832262] Key type ._llcrypt registered [ 3701.841214] Key type .llcrypt registered [ 3702.075125] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3703.769333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.810471] alg: No test for adler32 (adler32-zlib) [ 3705.338851] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3705.948188] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3707.735326] Key type lgssc registered [ 3708.947360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3719.156295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3719.189301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.794427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.592799] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3730.920530] LustreError: 120882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.945604] LustreError: 120882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3736.033228] LustreError: 120883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.877414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.477372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3742.042587] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3750.150258] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3756.216342] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3763.633502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3763.978635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.083716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3772.278218] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3781.224444] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3789.404418] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3793.972886] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3793.979184] Lustre: Skipped 2 previous similar messages [ 3797.557892] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3797.602307] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3799.003559] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3799.008958] Lustre: Skipped 2 previous similar messages [ 3803.676769] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3803.681195] Lustre: Skipped 1 previous similar message [ 3808.527056] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3808.546678] Lustre: Skipped 2 previous similar messages [ 3811.115951] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp [ 3824.611622] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.30/24@tcp in log lustre-MDT0000 [ 3824.633153] Lustre: Skipped 2 previous similar messages [ 3833.029189] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 3833.042869] Lustre: Skipped 1 previous similar message [ 3843.040512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.043366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.051843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.068195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.077949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.102779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.187760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3849.193310] Lustre: Skipped 2 previous similar messages [ 3854.309908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.324027] Lustre: Skipped 1 previous similar message [ 3856.352907] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3856.762446] Lustre: server umount lustre-OST0000 complete [ 3864.546105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.577234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3864.594325] Lustre: Skipped 1 previous similar message [ 3867.494411] Lustre: server umount lustre-MDT0000 complete [ 3869.675928] LustreError: 120882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.706539] LustreError: 120882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3872.230652] LustreError: 120864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950842 with bad export cookie 6485578910507174927 [ 3872.234105] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3872.237687] LustreError: 120864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3872.482081] Lustre: server umount lustre-MDT0001 complete [ 3885.375205] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3888.766460] Key type lgssc unregistered [ 3889.117631] LNet: 125742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3889.126559] LNetError: 125742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3889.143664] LNet: Removed LNI 192.168.201.130@tcp [ 3890.099304] Key type .llcrypt unregistered [ 3890.103521] Key type ._llcrypt unregistered [ 3927.353585] Key type ._llcrypt registered [ 3927.356597] Key type .llcrypt registered [ 3927.534345] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3929.579185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3929.617639] alg: No test for adler32 (adler32-zlib) [ 3930.809796] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 3931.052700] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3932.735563] Key type lgssc registered [ 3933.852449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3943.345081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3943.394890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.262776] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3945.281388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.328638] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.30/24@tcp [ 3951.316728] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3954.662545] LustreError: 127303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.686633] LustreError: 127303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3959.794380] LustreError: 127304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.203392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.702858] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3963.763140] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 3963.885945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3970.469041] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3980.204267] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3987.699773] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3996.468647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.934579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.461556] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4014.391721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4016.010841] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4029.921090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.946023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4029.951452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.955649] Lustre: Skipped 1 previous similar message [ 4032.920984] Lustre: server umount lustre-OST0000 complete [ 4040.162067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.185328] Lustre: Skipped 1 previous similar message [ 4040.194137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.057885] Lustre: server umount lustre-MDT0000 complete [ 4045.305275] LustreError: 127300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.327340] LustreError: 127300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4047.168575] LustreError: 127283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951017 with bad export cookie 14112466827538231312 [ 4047.178596] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4047.183976] LustreError: 127283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4047.541876] Lustre: server umount lustre-MDT0001 complete [ 4059.521887] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4063.114742] Key type lgssc unregistered [ 4063.460111] LNet: 130897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4063.467940] LNetError: 130897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4063.489221] LNet: Removed LNI 192.168.201.130@tcp [ 4064.490932] Key type .llcrypt unregistered [ 4064.492730] Key type ._llcrypt unregistered [ 4094.468459] Key type ._llcrypt registered [ 4094.471438] Key type .llcrypt registered [ 4094.566731] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4109.960259] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4111.454291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4111.549824] alg: No test for adler32 (adler32-zlib) [ 4112.553359] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 4112.804285] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4114.519222] Key type lgssc registered [ 4115.932677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4157.186452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4157.212968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.439841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4158.460116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4158.522717] Lustre: lustre-MDT0000: new disk, initializing [ 4158.640838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.677325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4164.200339] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4176.658455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.892696] Lustre: 134300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4177.007602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4177.024683] Lustre: Skipped 1 previous similar message [ 4177.163725] Lustre: lustre-MDT0001: new disk, initializing [ 4177.333111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.375790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4177.390225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4182.651778] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4192.304343] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4197.879854] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4204.385383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.679331] Lustre: lustre-OST0000: new disk, initializing [ 4204.682943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4204.694377] Lustre: 136272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4204.809626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.542134] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4214.362210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4214.382629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4214.598674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4221.510612] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.934795] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4227.284421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4233.239723] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4233.722722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4240.354082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4240.357070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.396628] Lustre: Skipped 1 previous similar message [ 4240.400855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.487715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.512363] Lustre: Skipped 1 previous similar message [ 4250.602543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4250.628251] Lustre: Skipped 1 previous similar message [ 4251.103569] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4251.291986] Lustre: server umount lustre-OST0000 complete [ 4259.299992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4259.305922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.319232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.326366] Lustre: Skipped 1 previous similar message [ 4260.838244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.670548] Lustre: server umount lustre-MDT0000 complete [ 4267.868787] LustreError: 134292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951238 with bad export cookie 9026446975586008402 [ 4267.874881] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.881139] LustreError: 134292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4268.273497] Lustre: server umount lustre-MDT0001 complete [ 4277.731967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.354714] LustreError: 138487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.381513] LustreError: 138487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4278.513958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.527791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4283.871781] LustreError: 138488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.996394] LustreError: 138487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.665609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.138948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4298.904667] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4309.365515] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4317.061755] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4324.764934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.104286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.430364] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4338.978435] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4344.274142] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4349.980890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4354.889461] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4355.133768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4365.534835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.790630] Lustre: lustre-OST0001: new disk, initializing [ 4365.793934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4365.805215] Lustre: 141894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4365.896814] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4371.504667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4371.517239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4371.604248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4372.974405] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4383.462428] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4383.853285] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4389.343947] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4389.614798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4398.081808] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e25451e6000.ost_server_uuid 50 [ 4400.152489] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e25451e6000.ost_server_uuid in IDLE state after 0 sec [ 4412.897066] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4412.903486] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.913501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4412.924641] Lustre: Skipped 2 previous similar messages [ 4417.592668] Lustre: server umount lustre-OST0001 complete [ 4418.037553] LustreError: 140308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.055917] LustreError: 140308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4423.137425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.141320] LustreError: 142192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.143849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4423.143858] Lustre: Skipped 1 previous similar message [ 4423.169418] Lustre: Skipped 2 previous similar messages [ 4423.208543] LustreError: 142192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4429.082580] Lustre: server umount lustre-OST0000 complete [ 4433.378962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.388511] Lustre: Skipped 1 previous similar message [ 4439.544830] Lustre: server umount lustre-MDT0000 complete [ 4443.636690] LustreError: 138488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.671728] LustreError: 138488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4444.443227] LustreError: 138468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951415 with bad export cookie 9026446975586009284 [ 4444.476113] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4444.843512] Lustre: server umount lustre-MDT0001 complete [ 4474.887959] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4478.609860] Key type lgssc unregistered [ 4478.986990] LNet: 144924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4479.006995] LNetError: 144924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4479.018931] LNet: Removed LNI 192.168.201.130@tcp [ 4480.300609] Key type .llcrypt unregistered [ 4480.311297] Key type ._llcrypt unregistered [ 4530.719838] Key type ._llcrypt registered [ 4530.731926] Key type .llcrypt registered [ 4531.024511] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4553.217715] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4554.757842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4554.891505] alg: No test for adler32 (adler32-zlib) [ 4556.175738] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 4556.380755] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4558.103185] Key type lgssc registered [ 4559.438482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4594.081487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.138065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.452036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4595.488583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4595.574861] Lustre: lustre-MDT0000: new disk, initializing [ 4595.664605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.693650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4600.797444] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4613.272827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.402283] Lustre: 149079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4613.484707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4613.488735] Lustre: Skipped 1 previous similar message [ 4613.549943] Lustre: lustre-MDT0001: new disk, initializing [ 4613.628803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4613.663120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4613.678828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4620.750413] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4631.010803] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4637.481681] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4644.361306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.615694] Lustre: lustre-OST0000: new disk, initializing [ 4644.618836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4644.624384] Lustre: 151048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4644.705609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.807430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4652.668547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4652.685967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4652.898125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4662.656727] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4666.720239] Lustre: Failing over lustre-OST0000 [ 4666.921129] Lustre: server umount lustre-OST0000 complete [ 4684.768992] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951639/real 1786951639] req@00000000df2112b9 x1873754484409344/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786951655 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4684.771213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.808022] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4685.856377] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951639/real 1786951639] req@0000000029580b92 x1873754484410496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786951655 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4685.887241] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4687.261233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4687.609199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.628296] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4690.003795] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951644/real 1786951644] req@00000000c2a4affe x1873754484411520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786951660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4694.725915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4694.758150] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951649/real 1786951649] req@000000008ef6400d x1873754484412672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786951665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4694.790578] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4694.801264] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4696.327158] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4696.330769] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4699.150813] Lustre: Failing over lustre-MDT0000 [ 4699.410658] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951654/real 1786951654] req@0000000076a4a883 x1873754484413824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786951670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4699.432736] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4699.475562] Lustre: server umount lustre-MDT0000 complete [ 4700.134312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.155828] Lustre: Skipped 1 previous similar message [ 4705.261152] LustreError: 150242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.278287] LustreError: 150242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4706.543961] LustreError: 149087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.369270] LustreError: 149091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.395882] LustreError: 149091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4715.488693] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951670/real 1786951670] req@0000000003b5d6a8 x1873754484423424/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786951686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4715.492935] LustreError: 149090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.541747] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.588119] LustreError: 149090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4718.599252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.609043] LustreError: 151056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.641364] LustreError: 151056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4725.727911] LustreError: 147302:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d9cf7a16 x1873754484428928/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4726.077359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.155586] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4727.017498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4731.394571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4731.404592] Lustre: Skipped 1 previous similar message [ 4731.417744] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4731.473837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4731.534806] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4741.614940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.635511] Lustre: Skipped 1 previous similar message [ 4741.647099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4746.724780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4746.731586] Lustre: Skipped 1 previous similar message [ 4747.388534] Lustre: server umount lustre-OST0000 complete [ 4751.841090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.858870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4751.863481] Lustre: Skipped 1 previous similar message [ 4751.865737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4751.865745] Lustre: Skipped 1 previous similar message [ 4756.963410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.974028] Lustre: Skipped 2 previous similar messages [ 4757.471404] Lustre: server umount lustre-MDT0000 complete [ 4761.526672] LustreError: 151047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951732 with bad export cookie 15394106686184920327 [ 4761.528529] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.535400] LustreError: 151047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4762.008025] Lustre: server umount lustre-MDT0001 complete [ 4774.406625] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4777.713355] Key type lgssc unregistered [ 4777.992321] LNet: 154708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4778.008258] LNetError: 154708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4779.045849] LNet: Removed LNI 192.168.201.130@tcp [ 4780.196607] Key type .llcrypt unregistered [ 4780.203925] Key type ._llcrypt unregistered [ 4821.850453] Key type ._llcrypt registered [ 4821.857738] Key type .llcrypt registered [ 4822.076524] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4824.662459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4825.199458] alg: No test for adler32 (adler32-zlib) [ 4826.764774] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 4827.038519] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4828.723193] Key type lgssc registered [ 4830.285390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4842.019864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4842.038708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.490367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.117688] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4853.731298] LustreError: 156232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.737614] LustreError: 156232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4857.625707] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4858.861407] LustreError: 156233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.954868] LustreError: 156261:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4861.964461] LustreError: 156261:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4861.971735] LustreError: 156261:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4862.329980] Lustre: server umount lustre-MDT0000 complete [ 4879.758732] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4890.581876] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4931.779528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.037948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4932.061523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4932.147184] Lustre: lustre-MDT0000: new disk, initializing [ 4932.201426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.213268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4937.748840] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4949.509268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.672527] Lustre: 159562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4949.750856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4949.758610] Lustre: Skipped 1 previous similar message [ 4949.825243] Lustre: lustre-MDT0001: new disk, initializing [ 4949.933174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4949.976361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4949.993900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4954.322044] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4964.149955] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4971.177695] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4979.966587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.562558] Lustre: lustre-OST0000: new disk, initializing [ 4980.583163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4980.601972] Lustre: 161544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4980.845770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.632720] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4990.046126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4990.066659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4990.268072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4998.371700] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.609256] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5005.472929] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:32:54 (1786951974) [ 5246.432604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.438070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.461844] Lustre: Skipped 2 previous similar messages [ 5246.488441] Lustre: Skipped 2 previous similar messages [ 5249.893482] Lustre: server umount lustre-MDT0000 complete [ 5251.558665] LustreError: 159569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.586153] LustreError: 159569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5254.326875] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952225 with bad export cookie 449415344364849684 [ 5254.334353] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.358718] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5256.673856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.675683] LustreError: 159571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5256.703976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5259.679845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5260.870972] Lustre: server umount lustre-MDT0001 complete [ 5271.917582] Lustre: server umount lustre-OST0000 complete [ 5284.898512] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5295.747214] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5334.868194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.140575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5335.167422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5335.279956] Lustre: lustre-MDT0000: new disk, initializing [ 5335.380985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.393563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5340.426409] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5351.473357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.591706] Lustre: 166202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5351.654666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5351.657931] Lustre: Skipped 1 previous similar message [ 5351.733362] Lustre: lustre-MDT0001: new disk, initializing [ 5351.852729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5351.926289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5351.948672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5356.935777] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5367.277654] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5372.607272] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5379.728337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.975251] Lustre: lustre-OST0000: new disk, initializing [ 5379.978811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5379.985738] Lustre: 168174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5380.063060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.581427] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5388.892111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5388.904414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5389.027704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5396.372985] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5401.122471] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5401.359729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5406.659958] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5406.865399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5409.764541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5409.767403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.769584] LustreError: Skipped 1 previous similar message [ 5409.770724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5409.806725] Lustre: Skipped 1 previous similar message [ 5414.882790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5414.888856] Lustre: Skipped 2 previous similar messages [ 5415.622817] Lustre: server umount lustre-OST0000 complete [ 5420.006738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.030506] Lustre: Skipped 1 previous similar message [ 5425.122728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.137070] Lustre: Skipped 3 previous similar messages [ 5425.805841] Lustre: server umount lustre-MDT0000 complete [ 5429.508581] LustreError: 166195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952400 with bad export cookie 449415344365008514 [ 5429.510878] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5429.532500] LustreError: 166195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5429.878857] Lustre: server umount lustre-MDT0001 complete [ 5437.635949] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:40:07 (1786952407) [ 5457.504071] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5469.922741] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5517.973492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.106526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5518.122587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5518.170274] Lustre: lustre-MDT0000: new disk, initializing [ 5518.215758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.243255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5523.497973] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5534.225387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.357404] Lustre: 172929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5534.712921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5539.485356] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5549.513513] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5556.349451] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5564.972536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.211384] Lustre: lustre-OST0000: new disk, initializing [ 5565.213941] Lustre: Skipped 1 previous similar message [ 5565.219452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5565.222878] Lustre: Skipped 2 previous similar messages [ 5565.228091] Lustre: 174910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5571.118180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5571.137399] Lustre: Skipped 1 previous similar message [ 5571.155692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5571.281506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5572.371262] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5583.140917] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5592.033957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5592.041134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.049676] LustreError: Skipped 1 previous similar message [ 5592.050473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5592.070646] Lustre: Skipped 1 previous similar message [ 5596.719606] Lustre: server umount lustre-OST0000 complete [ 5601.248541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5601.255173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.845836] Lustre: server umount lustre-MDT0000 complete [ 5607.393903] LustreError: 174161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.419695] LustreError: 174161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5610.840646] LustreError: 172922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952581 with bad export cookie 449415344365009396 [ 5610.842112] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.852747] LustreError: 172922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5611.325818] Lustre: server umount lustre-MDT0001 complete [ 5621.024599] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:43:09 (1786952589) [ 5638.419719] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5649.514530] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5694.611901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.175403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5695.271049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5695.395362] Lustre: lustre-MDT0000: new disk, initializing [ 5695.720504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5695.729989] Lustre: Skipped 2 previous similar messages [ 5695.768312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5702.665820] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5716.149825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.484464] Lustre: 179244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.987529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5722.685860] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5734.708996] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5740.776746] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.826507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.093532] Lustre: 181221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5756.196370] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5757.991705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5758.103556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5769.468885] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5783.520805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5783.523887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.576727] Lustre: Skipped 1 previous similar message [ 5783.599662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.601583] Lustre: Skipped 4 previous similar messages [ 5785.543722] Lustre: server umount lustre-OST0000 complete [ 5793.764358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.777242] LustreError: Skipped 1 previous similar message [ 5796.290707] Lustre: server umount lustre-MDT0000 complete [ 5799.399961] LustreError: 179254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.427568] LustreError: 179254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5801.038766] LustreError: 179235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952771 with bad export cookie 449415344365010481 [ 5801.041887] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.056284] LustreError: 179235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5801.528494] Lustre: server umount lustre-MDT0001 complete [ 5814.398520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5817.518521] Key type lgssc unregistered [ 5817.820289] LNet: 183082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.831796] LNetError: 183082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.846796] LNet: Removed LNI 192.168.201.130@tcp [ 5818.660833] Key type .llcrypt unregistered [ 5818.662420] Key type ._llcrypt unregistered [ 5847.527391] Key type ._llcrypt registered [ 5847.533421] Key type .llcrypt registered [ 5847.735923] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5849.708434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5849.737087] alg: No test for adler32 (adler32-zlib) [ 5850.853642] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 5851.180677] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5852.881115] Key type lgssc registered [ 5854.296506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5863.516800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5863.528741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.131527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.644456] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5874.664231] LustreError: 184499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.692822] LustreError: 184499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5879.796945] LustreError: 184500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.779472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.312867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5886.416171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5893.995152] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5900.468551] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5908.784996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5909.221867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.270113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5916.572321] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5925.996464] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.336713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.371134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.381684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.392249] Lustre: Skipped 1 previous similar message [ 5951.460530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5951.467327] Lustre: Skipped 1 previous similar message [ 5952.345735] Lustre: server umount lustre-OST0000 complete [ 5958.113353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5958.120672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.128512] Lustre: Skipped 1 previous similar message [ 5958.134920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.698249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.700069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.732833] Lustre: Skipped 1 previous similar message [ 5963.317749] Lustre: server umount lustre-MDT0000 complete [ 5966.816491] LustreError: 184499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.846215] LustreError: 184499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5967.647153] LustreError: 184479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952938 with bad export cookie 16799250926577086376 [ 5967.656935] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5967.663233] LustreError: 184479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5968.023510] Lustre: server umount lustre-MDT0001 complete [ 5979.731475] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5983.402455] Key type lgssc unregistered [ 5983.831591] LNet: 188064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5983.858520] LNetError: 188064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5983.896191] LNet: Removed LNI 192.168.201.130@tcp [ 5984.751140] Key type .llcrypt unregistered [ 5984.752728] Key type ._llcrypt unregistered [ 6010.662397] Key type ._llcrypt registered [ 6010.664701] Key type .llcrypt registered [ 6010.773618] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6012.121650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6012.470588] alg: No test for adler32 (adler32-zlib) [ 6013.693523] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 6013.855751] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 6015.519898] Key type lgssc registered [ 6017.003736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.014625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.028360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.632226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.316537] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6037.989047] LustreError: 189480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.025353] LustreError: 189480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6043.105846] LustreError: 189481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.476701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.902930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6049.419867] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6057.162714] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6063.430847] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.814232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.035680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6074.179056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6076.877538] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6084.709589] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6093.798092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.803888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6093.825258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.839893] Lustre: Skipped 1 previous similar message [ 6093.866590] Lustre: Skipped 1 previous similar message [ 6096.671549] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 6098.930394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6098.936256] Lustre: Skipped 1 previous similar message [ 6099.804218] Lustre: server umount lustre-OST0000 complete [ 6107.621994] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e2547360800.ost_server_uuid 50 [ 6109.324863] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6123.374283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.389623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.642607] LustreError: 189480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.662160] LustreError: 189480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6129.751444] Lustre: server umount lustre-MDT0000 complete [ 6133.898984] LustreError: 189462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953104 with bad export cookie 12530941460994508573 [ 6133.915497] LustreError: 189462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6133.915834] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.554919] Lustre: server umount lustre-MDT0001 complete [ 6146.039328] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:51:55 (1786953115) [ 6154.125047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.719926] LustreError: 193400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.795370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.852756] LustreError: 193401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.025961] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6170.090880] LustreError: 193400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.121558] LustreError: 193400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6170.295133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.760506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6175.718035] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6185.242478] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.389564] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6199.214927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.623306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.690606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6206.484838] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6215.646460] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.460706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.695130] Lustre: lustre-OST0001: new disk, initializing [ 6223.703504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6223.714516] Lustre: 196241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6223.848675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6231.185334] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6233.719323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6233.735323] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6233.885226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6241.332653] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6246.884049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.902789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.927174] Lustre: Skipped 1 previous similar message [ 6246.949608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.959630] Lustre: Skipped 3 previous similar messages [ 6247.906706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6250.627514] Lustre: server umount lustre-OST0000 complete [ 6252.266515] LustreError: 195224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.102031] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6258.362459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6263.358388] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6263.585616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6280.163019] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6280.163513] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.172659] LustreError: Skipped 1 previous similar message [ 6280.202133] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6280.206234] Lustre: Skipped 2 previous similar messages [ 6280.232301] Lustre: Skipped 5 previous similar messages [ 6285.283344] LustreError: 195222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.321542] LustreError: 195222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6285.436516] Lustre: server umount lustre-OST0001 complete [ 6290.403822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.418409] Lustre: Skipped 1 previous similar message [ 6296.057754] Lustre: server umount lustre-MDT0000 complete [ 6300.277035] LustreError: 196251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953270 with bad export cookie 12530941460994509644 [ 6300.289110] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.298538] LustreError: 196251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.647763] Lustre: server umount lustre-MDT0001 complete [ 6327.484096] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:54:57 (1786953297) [ 6334.513620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.859156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6334.924860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6335.195100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.740622] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6348.895341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.026879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6349.070964] Lustre: 199683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6355.227598] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6363.823081] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.566956] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6379.967319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.171532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6387.146704] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6387.847068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6395.159829] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.253173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.364753] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6402.613967] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6402.624959] Lustre: Skipped 2 previous similar messages [ 6409.839899] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6412.483638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6419.906713] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6423.525819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.536655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.563270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.575738] Lustre: Skipped 4 previous similar messages [ 6429.499964] Lustre: server umount lustre-OST0000 complete [ 6430.952656] LustreError: 201531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.098605] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.106617] LustreError: Skipped 1 previous similar message [ 6448.111263] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.128164] Lustre: Skipped 1 previous similar message [ 6452.107707] Lustre: server umount lustre-OST0001 complete [ 6456.821037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6456.826768] Lustre: Skipped 6 previous similar messages [ 6462.522694] Lustre: server umount lustre-MDT0000 complete [ 6466.647303] LustreError: 202566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953437 with bad export cookie 12530941460994511044 [ 6466.657313] LustreError: 202566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6466.658483] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.986752] Lustre: server umount lustre-MDT0001 complete [ 6497.927234] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:57:47 (1786953467) [ 6514.710315] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 6526.779859] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6569.088756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.279778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6569.296395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6569.351902] Lustre: lustre-MDT0000: new disk, initializing [ 6569.413244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.428596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6574.015420] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6585.284694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.453233] Lustre: 207776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6585.507962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6585.510689] Lustre: Skipped 1 previous similar message [ 6585.579772] Lustre: lustre-MDT0001: new disk, initializing [ 6585.723509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6585.746438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6591.385252] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6602.688398] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6609.235387] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6616.611182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6616.787466] Lustre: lustre-OST0000: new disk, initializing [ 6616.791980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6616.798426] Lustre: 209744:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6622.535829] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6623.298968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6623.322224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6623.515324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6632.942842] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.673179] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6639.152776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6645.008622] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6645.274943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6649.315946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.320114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.351938] LustreError: Skipped 2 previous similar messages [ 6649.368367] Lustre: Skipped 3 previous similar messages [ 6649.380609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6649.396556] Lustre: Skipped 3 previous similar messages [ 6654.267293] Lustre: server umount lustre-OST0000 complete [ 6664.940967] Lustre: server umount lustre-MDT0000 complete [ 6669.662957] LustreError: 207769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953640 with bad export cookie 12530941460994512836 [ 6669.683288] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6670.389706] Lustre: server umount lustre-MDT0001 complete [ 6680.391035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.844658] LustreError: 211962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.860823] LustreError: 211962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6685.812977] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6695.843391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.001800] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6709.267798] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6715.665875] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.488742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6730.448836] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6738.766619] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6744.444520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6744.658723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6749.290562] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6749.535160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6751.715067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6751.720944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.739176] Lustre: Skipped 3 previous similar messages [ 6757.842078] Lustre: server umount lustre-OST0000 complete [ 6766.559117] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6771.970078] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6801.997657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.478121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6805.521225] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6809.121855] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6818.213612] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.891411] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6820.909606] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6820.937326] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6825.955131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6825.969432] LustreError: Skipped 2 previous similar messages [ 6825.974934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.995372] Lustre: Skipped 8 previous similar messages [ 6838.240841] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6838.425611] Lustre: server umount lustre-OST0000 complete [ 6849.595797] Lustre: server umount lustre-MDT0000 complete [ 6854.626925] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953825 with bad export cookie 12530941460994513718 [ 6854.631133] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.655279] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6855.427518] Lustre: server umount lustre-MDT0001 complete [ 6865.135147] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:03:54 (1786953834) [ 6872.193932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.910121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.916617] Lustre: Skipped 6 previous similar messages [ 6877.401717] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6886.228287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.376421] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6900.975084] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.948452] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6916.060132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.027669] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6931.633429] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6936.559051] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6936.813481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.411797] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6941.662315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6948.172811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.396602] Lustre: lustre-OST0001: new disk, initializing [ 6948.400616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6948.410922] Lustre: 221314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6956.133568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6956.149434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6956.250182] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6956.310052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6969.586086] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6975.936479] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6976.403722] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6982.170905] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6982.380179] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6987.232442] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.234985] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.248913] LustreError: Skipped 2 previous similar messages [ 6987.278508] Lustre: Skipped 6 previous similar messages [ 6990.753791] Lustre: server umount lustre-OST0001 complete [ 6992.363114] LustreError: 221663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.378697] LustreError: 221663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6997.617713] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6997.836321] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7002.949371] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7003.308801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7034.207288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.986246] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7040.017887] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7040.033804] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7042.270169] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7050.781326] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7052.265346] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7060.229931] Lustre: server umount lustre-OST0001 complete [ 7084.466995] Lustre: server umount lustre-OST0000 complete [ 7091.169677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7091.175649] Lustre: Skipped 14 previous similar messages [ 7095.450996] Lustre: server umount lustre-MDT0000 complete [ 7099.741536] LustreError: 217926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954070 with bad export cookie 12530941460994514684 [ 7099.744685] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.756275] LustreError: 217926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7114.207199] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7114.810416] Lustre: server umount lustre-MDT0001 complete [ 7142.567936] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:08:32 (1786954112) [ 7149.685338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.142505] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7150.202617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7155.672789] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7165.127191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.266694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7165.339293] Lustre: 226561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7170.504523] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7177.442868] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.925928] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7192.667965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.914504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7200.686075] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7208.195537] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7218.267988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.347624] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7225.589665] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7234.173355] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.036760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7241.168986] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7241.525611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7246.686316] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7246.982732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7255.712739] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e25451e5800.ost_server_uuid 50 [ 7258.022063] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e25451e5800.ost_server_uuid in IDLE state after 0 sec [ 7259.495781] Lustre: Permanently deactivating lustre-OST0001 [ 7259.518960] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7264.893448] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7265.237570] Lustre: Permanently reactivating lustre-OST0001 [ 7265.308324] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7265.322944] Lustre: Skipped 2 previous similar messages [ 7275.110715] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.122553] Lustre: Skipped 6 previous similar messages [ 7275.463543] Lustre: server umount lustre-OST0001 complete [ 7286.807742] Lustre: server umount lustre-OST0000 complete [ 7293.929861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7293.943730] LustreError: Skipped 2 previous similar messages [ 7295.457773] LustreError: 231143:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7302.249218] Lustre: server umount lustre-MDT0000 complete [ 7306.660385] LustreError: 227314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954277 with bad export cookie 12530941460994515972 [ 7306.663223] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.673082] LustreError: 227314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7307.415324] Lustre: server umount lustre-MDT0001 complete [ 7337.928896] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:11:47 (1786954307) [ 7346.704855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.047165] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7347.078113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7347.092210] Lustre: Skipped 2 previous similar messages [ 7352.289059] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7361.479348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.616534] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7361.650068] Lustre: 233268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7366.588774] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7374.424412] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7381.138582] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7388.114700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7388.360499] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7388.373444] Lustre: Permanently deactivating lustre-OST0000 [ 7388.381314] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7388.540650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.552657] Lustre: Skipped 10 previous similar messages [ 7394.384685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7394.387579] Lustre: Skipped 1 previous similar message [ 7395.165819] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7404.338274] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7412.183062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7412.293319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7419.528428] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7419.701149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7427.004046] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.893710] Lustre: Permanently reactivating lustre-OST0000 [ 7428.942844] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7428.952628] Lustre: Skipped 2 previous similar messages [ 7429.612042] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7429.621531] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 0000000058ab483f, cur 1786954400 deadline 1786954395 last 1786954365 [ 7429.670147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7461.343717] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7461.438990] Lustre: server umount lustre-OST0001 complete [ 7471.493614] Lustre: server umount lustre-OST0000 complete [ 7481.804103] Lustre: server umount lustre-MDT0000 complete [ 7485.978557] LustreError: 234012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954456 with bad export cookie 12530941460994517708 [ 7485.981072] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.996945] LustreError: 234012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7486.386704] Lustre: server umount lustre-MDT0001 complete [ 7498.572442] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 7502.955658] Key type lgssc unregistered [ 7503.530664] LNet: 238161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.549846] LNetError: 238161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.577707] LNet: Removed LNI 192.168.201.130@tcp [ 7505.488194] Key type .llcrypt unregistered [ 7505.491507] Key type ._llcrypt unregistered [ 7532.507311] Key type ._llcrypt registered [ 7532.510716] Key type .llcrypt registered [ 7532.653211] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7534.622387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7534.658214] alg: No test for adler32 (adler32-zlib) [ 7535.872756] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 7536.352679] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 7538.079254] Key type lgssc registered [ 7539.606388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7550.252492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7550.270488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.967450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7557.281530] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7562.209789] LustreError: 239627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.229129] LustreError: 239627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7567.124636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.297810] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7567.334552] LustreError: 239626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.351346] Lustre: 239615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7567.375589] Lustre: Permanently deactivating lustre-MDT0001 [ 7567.407979] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7567.853599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7572.464449] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7573.538467] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7579.897645] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7585.319908] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7592.691861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.059855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.162069] LustreError: 241474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.180898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7600.854291] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7602.679401] LustreError: 241472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.785036] LustreError: 241474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7609.505823] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7612.911990] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7616.888680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7617.162592] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7622.644739] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7622.647253] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7623.429244] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7631.185645] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7635.671569] Lustre: Permanently reactivating lustre-MDT0001 [ 7635.699670] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7635.706044] Lustre: Skipped 2 previous similar messages [ 7637.885187] Lustre: lustre-MDT0001: Client b2c6d099-8d8f-4b69-8bd6-264f16d18f6e (at 192.168.201.30@tcp) reconnecting [ 7641.638110] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.648063] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7641.654379] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7641.670632] LustreError: 243310:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7641.672176] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7641.690349] LustreError: 243310:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7641.713512] LustreError: 243310:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.130@tcp: cfg command failed: rc = -108 [ 7641.734252] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7641.734252] [ 7641.748634] LustreError: 239618:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7644.648656] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7646.434291] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7652.912435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7653.239480] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7658.634267] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7658.844219] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7661.363850] Lustre: Permanently deactivating lustre-MDT0001 [ 7661.394835] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7661.420115] Lustre: Skipped 2 previous similar messages [ 7668.450687] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7675.170211] Lustre: server umount lustre-MDT0000 complete [ 7675.368452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7675.374503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.403993] LustreError: 239627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.416034] LustreError: 239627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7677.923413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.944515] Lustre: Skipped 2 previous similar messages [ 7679.212847] LustreError: 240368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954649 with bad export cookie 12383396652276871547 [ 7679.220139] LustreError: 240368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7679.226790] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.596597] Lustre: server umount lustre-MDT0001 complete [ 7683.696652] Lustre: server umount lustre-OST0000 complete [ 7697.655939] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:17:47 (1786954667) [ 7710.662913] Lustre: server umount lustre-OST0001 complete [ 7718.194612] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 7728.290334] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7764.695195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.928838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7764.944660] Lustre: Skipped 2 previous similar messages [ 7764.988871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7765.140497] Lustre: lustre-MDT0000: new disk, initializing [ 7765.294705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.338734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7770.229157] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7781.950320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7782.062371] Lustre: 247889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7782.110932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7782.115202] Lustre: Skipped 1 previous similar message [ 7782.208090] Lustre: lustre-MDT0001: new disk, initializing [ 7782.305460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.360453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7782.377713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7786.772851] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7796.625117] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7804.038535] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7811.769280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7812.296351] Lustre: lustre-OST0000: new disk, initializing [ 7812.327609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7812.340672] Lustre: 249857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7812.489766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.893955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7819.927605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7820.088326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7820.322921] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7830.602437] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.069581] LustreError: 247897:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7843.120094] LustreError: 247897:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7843.158990] LustreError: 247896:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7843.212408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.462866] Lustre: lustre-OST0001: new disk, initializing [ 7843.466650] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7843.472245] Lustre: 250972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7850.049698] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7852.783243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7852.796624] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7852.922325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7853.175134] LustreError: 247896:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7862.525124] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7864.328368] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7868.386037] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.387208] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.402548] LustreError: Skipped 1 previous similar message [ 7868.404213] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7868.445385] Lustre: Skipped 1 previous similar message [ 7872.956432] Lustre: server umount lustre-OST0001 complete [ 7873.518932] LustreError: 249841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.560458] LustreError: 249841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7891.936978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7891.939954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.949398] LustreError: Skipped 1 previous similar message [ 7891.952379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.958900] Lustre: Skipped 1 previous similar message [ 7891.981452] Lustre: Skipped 2 previous similar messages [ 7893.984808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7893.989785] Lustre: Skipped 1 previous similar message [ 7895.552824] Lustre: server umount lustre-OST0000 complete [ 7899.111069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.116659] Lustre: Skipped 1 previous similar message [ 7904.225853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.240308] Lustre: Skipped 1 previous similar message [ 7905.230683] Lustre: server umount lustre-MDT0000 complete [ 7909.346602] LustreError: 249121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.369686] LustreError: 249121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7909.505805] LustreError: 249856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954880 with bad export cookie 12383396652276874361 [ 7909.508160] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7909.525511] LustreError: 249856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7915.920462] Lustre: server umount lustre-MDT0001 complete [ 7926.927773] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 7929.780839] Key type lgssc unregistered [ 7930.017610] LNet: 253008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7930.074419] LNetError: 253008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7930.099241] LNet: Removed LNI 192.168.201.130@tcp [ 7931.125682] Key type .llcrypt unregistered [ 7931.128044] Key type ._llcrypt unregistered [ 7973.217508] Key type ._llcrypt registered [ 7973.220072] Key type .llcrypt registered [ 7973.321294] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7974.948898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.986748] alg: No test for adler32 (adler32-zlib) [ 7976.012247] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 7976.324728] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 7978.031243] Key type lgssc registered [ 7979.384860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7987.260478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7987.271210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.702532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7988.741324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7989.027091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.006001] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8003.488231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.693248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8003.733321] Lustre: 255218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.131903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8008.733375] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8017.112024] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8023.049493] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8029.947689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.171669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8030.371781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.637393] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8039.025000] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8043.473557] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8095.224967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8095.232346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.247573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.841786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.851587] Lustre: Skipped 2 previous similar messages [ 8101.171769] Lustre: server umount lustre-OST0000 complete [ 8105.690620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8120.011340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.413126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.540317] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8125.554432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8125.568733] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8126.915911] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8135.918921] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8177.121613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8177.123829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.130093] LustreError: Skipped 2 previous similar messages [ 8177.138861] Lustre: Skipped 1 previous similar message [ 8177.142454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.158098] Lustre: Skipped 1 previous similar message [ 8182.198215] Lustre: server umount lustre-OST0000 complete [ 8187.367887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.378664] Lustre: Skipped 1 previous similar message [ 8187.390445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.394567] Lustre: Skipped 1 previous similar message [ 8192.481723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.487797] Lustre: Skipped 1 previous similar message [ 8200.159319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8200.586754] Lustre: server umount lustre-MDT0000 complete [ 8205.282765] LustreError: 257081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955175 with bad export cookie 497895387963061912 [ 8205.289995] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8205.293136] LustreError: 257081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8205.944827] Lustre: server umount lustre-MDT0001 complete [ 8220.482406] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8223.642740] Key type lgssc unregistered [ 8224.155504] LNet: 260648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8224.168217] LNetError: 260648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8224.207462] LNet: Removed LNI 192.168.201.130@tcp [ 8225.355673] Key type .llcrypt unregistered [ 8225.358099] Key type ._llcrypt unregistered [ 8251.584732] Key type ._llcrypt registered [ 8251.586656] Key type .llcrypt registered [ 8251.676794] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8253.212045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8253.273762] alg: No test for adler32 (adler32-zlib) [ 8254.408357] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 8254.651935] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8256.383681] Key type lgssc registered [ 8257.820129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8266.693303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8266.710584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.312524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.922845] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8277.479358] LustreError: 262066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8277.503342] LustreError: 262066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8281.955940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.414915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8287.303077] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8294.732044] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.319922] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8307.038177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8307.486742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.627166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8314.737616] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8322.578954] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8348.640276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8348.658505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.671157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.681619] Lustre: Skipped 1 previous similar message [ 8353.113459] Lustre: server umount lustre-OST0000 complete [ 8358.882758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.887913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.909789] Lustre: Skipped 1 previous similar message [ 8358.928034] Lustre: Skipped 2 previous similar messages [ 8363.572927] Lustre: server umount lustre-MDT0000 complete [ 8364.005238] LustreError: 262062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8364.029321] LustreError: 262062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8367.014780] LustreError: 262544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.637315] LustreError: 262047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955338 with bad export cookie 2089100376221203870 [ 8367.648222] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8367.651555] LustreError: 262047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8367.941955] Lustre: server umount lustre-MDT0001 complete [ 8380.922905] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8384.077672] Key type lgssc unregistered [ 8384.428704] LNet: 266381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8384.437629] LNetError: 266381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8384.450779] LNet: Removed LNI 192.168.201.130@tcp [ 8385.535448] Key type .llcrypt unregistered [ 8385.539411] Key type ._llcrypt unregistered [ 8403.675981] Key type ._llcrypt registered [ 8403.679676] Key type .llcrypt registered [ 8403.812759] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8404.997291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8405.169599] alg: No test for adler32 (adler32-zlib) [ 8406.212854] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 8406.514330] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8408.287340] Key type lgssc registered [ 8409.559728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8418.267179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8418.286666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.062654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8425.567956] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8429.543842] LustreError: 267607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.575675] LustreError: 267607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8434.673790] LustreError: 267606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.488209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.945439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8441.466388] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8449.783514] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8456.741256] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8465.042037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8465.434164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.542581] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8481.662523] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8490.046083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8505.320598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8505.331960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.348195] Lustre: Skipped 1 previous similar message [ 8505.356545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8505.375961] Lustre: Skipped 1 previous similar message [ 8505.848277] Lustre: server umount lustre-OST0000 complete [ 8510.433456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.465970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.473859] Lustre: Skipped 1 previous similar message [ 8515.555124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.561216] Lustre: Skipped 1 previous similar message [ 8516.268247] Lustre: server umount lustre-MDT0000 complete [ 8520.680543] LustreError: 267606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.704443] LustreError: 267606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8521.824582] LustreError: 267586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955492 with bad export cookie 10082097711624718056 [ 8521.832084] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8521.836885] LustreError: 267586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8522.237859] Lustre: server umount lustre-MDT0001 complete [ 8534.543551] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8538.314669] Key type lgssc unregistered [ 8538.660399] LNet: 271308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8538.671636] LNetError: 271308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8539.686452] LNet: Removed LNI 192.168.201.130@tcp [ 8540.584874] Key type .llcrypt unregistered [ 8540.587095] Key type ._llcrypt unregistered [ 8564.597404] Key type ._llcrypt registered [ 8564.606363] Key type .llcrypt registered [ 8564.826426] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8566.095718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8566.141340] alg: No test for adler32 (adler32-zlib) [ 8567.176676] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 8567.538780] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8569.271138] Key type lgssc registered [ 8570.486271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8578.891743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.906171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8580.379696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.313299] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8590.818847] LustreError: 272742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.856054] LustreError: 272742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8595.955303] LustreError: 272743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.347563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.055153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.427617] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8615.819549] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.152312] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8628.456727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.971359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.540515] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8643.953627] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8650.503454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8676.323954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.326912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.370871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.379846] Lustre: Skipped 1 previous similar message [ 8681.175413] Lustre: server umount lustre-OST0000 complete [ 8686.563871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.586125] Lustre: Skipped 1 previous similar message [ 8686.598286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.604023] Lustre: Skipped 1 previous similar message [ 8691.589269] Lustre: server umount lustre-MDT0000 complete [ 8691.681385] LustreError: 273472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.704637] LustreError: 273472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8696.213037] LustreError: 272723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955666 with bad export cookie 11216829907133989063 [ 8696.227124] LustreError: 272723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8696.232611] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8696.938453] Lustre: server umount lustre-MDT0001 complete [ 8710.369937] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8712.873391] Key type lgssc unregistered [ 8713.207416] LNet: 277054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8713.214934] LNetError: 277054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8713.231306] LNet: Removed LNI 192.168.201.130@tcp [ 8714.092797] Key type .llcrypt unregistered [ 8714.098167] Key type ._llcrypt unregistered [ 8734.828766] Key type ._llcrypt registered [ 8734.830601] Key type .llcrypt registered [ 8734.975316] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8736.524495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8736.605298] alg: No test for adler32 (adler32-zlib) [ 8737.769239] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 8738.130929] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8739.971488] Key type lgssc registered [ 8741.254664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8750.327629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8750.342143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.933969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.824215] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8762.356782] LustreError: 278292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.376237] LustreError: 278292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8766.005265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8766.357593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8771.520176] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8779.520593] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8785.640713] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8792.803822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8793.353052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.492552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8800.238974] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8807.852832] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8823.265925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8823.274604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.295931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8826.339021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.344182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8826.368621] Lustre: Skipped 1 previous similar message [ 8828.854510] Lustre: server umount lustre-OST0000 complete [ 8833.017136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8833.033392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.053284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.577417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.585935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.609126] Lustre: Skipped 1 previous similar message [ 8838.595635] Lustre: server umount lustre-MDT0000 complete [ 8841.698637] LustreError: 278304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.721623] LustreError: 278304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8842.497477] LustreError: 278259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955813 with bad export cookie 10389831802675078671 [ 8842.521147] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8842.905767] Lustre: server umount lustre-MDT0001 complete [ 8853.326373] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8856.457188] Key type lgssc unregistered [ 8856.723824] LNet: 281995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8856.736271] LNetError: 281995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8856.762533] LNet: Removed LNI 192.168.201.130@tcp [ 8857.553298] Key type .llcrypt unregistered [ 8857.556883] Key type ._llcrypt unregistered [ 8869.084403] Key type ._llcrypt registered [ 8869.089205] Key type .llcrypt registered [ 8869.202474] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8886.125846] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 8905.794769] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8906.972273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8907.050100] alg: No test for adler32 (adler32-zlib) [ 8908.204738] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 8908.598167] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8910.375239] Key type lgssc registered [ 8911.816579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8953.470638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8953.537192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.837933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8954.896307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8955.009510] Lustre: lustre-MDT0000: new disk, initializing [ 8955.132092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.159468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8959.634582] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8972.136764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.365959] Lustre: 285551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8972.478617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8972.482537] Lustre: Skipped 1 previous similar message [ 8972.575330] Lustre: lustre-MDT0001: new disk, initializing [ 8972.688429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8972.731657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8972.746615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8977.312302] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8987.173772] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8993.249762] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9000.452725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.839883] Lustre: lustre-OST0000: new disk, initializing [ 9000.852908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9000.861546] Lustre: 287520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9001.003851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9007.278132] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9010.242191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9010.257124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9010.445740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9017.221410] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9024.414665] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9024.686987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9030.380645] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9030.766451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9036.264429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.264429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9036.270910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9036.284276] Lustre: Skipped 1 previous similar message [ 9036.299310] Lustre: Skipped 1 previous similar message [ 9039.876465] Lustre: server umount lustre-OST0000 complete [ 9046.497085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.523270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9046.536096] Lustre: Skipped 1 previous similar message [ 9051.157048] Lustre: server umount lustre-MDT0000 complete [ 9056.410410] LustreError: 285544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956027 with bad export cookie 54732357859004118 [ 9056.452430] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9057.050562] Lustre: server umount lustre-MDT0001 complete [ 9068.829902] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:40:38 (1786956038) [ 9078.087770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.782097] LustreError: 289932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.806549] LustreError: 289932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9078.997711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.396771] LustreError: 289933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.306461] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9088.492024] LustreError: 289932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.614879] LustreError: 289933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9096.355607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.652870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9102.937115] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9112.834403] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9120.563608] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.168956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.669054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.457765] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9146.187415] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9162.721645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9162.725062] LustreError: Skipped 1 previous similar message [ 9162.749816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.762970] Lustre: Skipped 1 previous similar message [ 9162.769562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.798519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.808871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.825151] Lustre: Skipped 1 previous similar message [ 9165.825375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9167.709379] Lustre: server umount lustre-OST0000 complete [ 9173.472645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9173.488871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.506652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.660721] Lustre: server umount lustre-MDT0000 complete [ 9176.037744] LustreError: 289932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.060907] LustreError: 289932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9178.131920] LustreError: 292535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956148 with bad export cookie 54732357859005000 [ 9178.144230] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.149330] LustreError: 292535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9178.855371] Lustre: server umount lustre-MDT0001 complete [ 9190.680354] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 9193.698993] Key type lgssc unregistered [ 9193.954763] LNet: 293493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9193.964578] LNetError: 293493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9193.991469] LNet: Removed LNI 192.168.201.130@tcp [ 9195.038653] Key type .llcrypt unregistered [ 9195.041996] Key type ._llcrypt unregistered [ 9221.954776] Key type ._llcrypt registered [ 9221.962399] Key type .llcrypt registered [ 9222.106513] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9237.457311] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9238.831938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9238.945606] alg: No test for adler32 (adler32-zlib) [ 9240.090549] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 9240.374093] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9242.090150] Key type lgssc registered [ 9243.643435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9291.665903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9291.685955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.948889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9292.995864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9293.178535] Lustre: lustre-MDT0000: new disk, initializing [ 9293.257650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.309641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9297.220510] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9308.975488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.100960] Lustre: 297336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9309.173424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9309.179587] Lustre: Skipped 1 previous similar message [ 9309.280118] Lustre: lustre-MDT0001: new disk, initializing [ 9309.416778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9309.475363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9309.491160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9314.070728] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9322.754461] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9327.948424] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9335.531428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9335.846929] Lustre: lustre-OST2710: new disk, initializing [ 9335.850771] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9335.865672] Lustre: 299302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9335.974242] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9343.514737] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9343.569401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9343.580702] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9343.658267] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9354.814707] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9362.515776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9362.663560] Lustre: lustre-OST03e8: new disk, initializing [ 9362.669833] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9362.679771] Lustre: 300368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9362.808795] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9369.605652] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9370.275503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9370.284455] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9370.394723] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9376.751694] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9384.590352] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9384.844877] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9390.396747] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9390.711366] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9395.632573] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9395.800535] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9400.483239] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9400.811570] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9406.444286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.470148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9406.471329] Lustre: Skipped 2 previous similar messages [ 9406.484836] Lustre: Skipped 2 previous similar messages [ 9407.976428] Lustre: server umount lustre-MDT0000 complete [ 9412.412229] LustreError: 297329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956383 with bad export cookie 52045426225033734 [ 9412.423531] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9412.999823] Lustre: server umount lustre-MDT0001 complete [ 9428.383686] Lustre: server umount lustre-OST2710 complete [ 9442.360566] Lustre: server umount lustre-OST03e8 complete [ 9461.006155] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9471.433967] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9510.646152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9510.879276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9510.896886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9510.951524] Lustre: lustre-MDT0000: new disk, initializing [ 9511.033660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.047588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9516.180959] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9527.177646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.328821] Lustre: 305617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9527.386835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9527.389043] Lustre: Skipped 1 previous similar message [ 9527.450524] Lustre: lustre-MDT0001: new disk, initializing [ 9527.526036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9527.562265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9527.577628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9532.555936] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9542.157028] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9548.268241] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9555.379718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.658211] Lustre: lustre-OST0000: new disk, initializing [ 9555.663881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9555.674153] Lustre: 307585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9555.799919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.812664] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9563.214555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9563.231387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9563.374022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9571.643785] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9576.406874] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9576.581492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9581.643630] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9581.902314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9584.099973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9584.102425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.106324] LustreError: Skipped 1 previous similar message [ 9584.106927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9584.135125] Lustre: Skipped 2 previous similar messages [ 9589.218155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9589.231403] Lustre: Skipped 1 previous similar message [ 9590.304758] Lustre: server umount lustre-OST0000 complete [ 9599.457798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9599.463332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.483749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9599.494375] Lustre: Skipped 1 previous similar message [ 9599.502931] Lustre: Skipped 2 previous similar messages [ 9600.979205] Lustre: server umount lustre-MDT0000 complete [ 9604.577260] LustreError: 305627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.605023] LustreError: 305627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9605.070059] LustreError: 306412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956575 with bad export cookie 52045426225035414 [ 9605.076622] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.079193] LustreError: 306412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9605.550580] Lustre: server umount lustre-MDT0001 complete [ 9614.953334] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:49:44 (1786956584) [ 9616.788706] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9618.981166] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:49:48 (1786956588) [ 9623.105634] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9650.470301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.948660] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9651.014606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9651.366835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.781187] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9666.644158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.817410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9666.915052] Lustre: 311440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9672.651891] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9695.104634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9695.514838] Lustre: Denying initial registration attempt from nid 192.168.201.130@tcp, specified as failover [ 9695.531412] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9695.556563] LustreError: 312895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9695.573746] LustreError: 312895:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9695.588591] LustreError: 312895:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9695.688562] Lustre: server umount lustre-OST0000 complete [ 9695.691604] LustreError: 312895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9703.398277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9703.425192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.471988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.512701] Lustre: server umount lustre-MDT0000 complete [ 9713.428646] LustreError: 311432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956684 with bad export cookie 52045426225036289 [ 9713.441262] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9713.444028] LustreError: 311432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9713.913646] Lustre: server umount lustre-MDT0001 complete [ 9725.785578] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 9728.900824] Key type lgssc unregistered [ 9729.279664] LNet: 314020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9729.283884] LNetError: 314020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9729.299618] LNet: Removed LNI 192.168.201.130@tcp [ 9730.512799] Key type .llcrypt unregistered [ 9730.515380] Key type ._llcrypt unregistered [ 9744.255338] Key type ._llcrypt registered [ 9744.260534] Key type .llcrypt registered [ 9744.412893] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9745.372337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9745.393159] alg: No test for adler32 (adler32-zlib) [ 9746.439362] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 9746.813954] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9748.535161] Key type lgssc registered [ 9750.104748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9798.553679] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9808.222709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9808.252633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.562492] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9809.595347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9809.893854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.998924] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9824.278252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.475210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9824.544727] Lustre: 316786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9824.905054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9830.818646] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9840.299845] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9846.299523] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9866.793509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9867.056867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9867.267812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9873.778694] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9882.966562] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.178098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9886.192590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.215483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9886.688303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9886.699528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.729682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9891.686135] Lustre: server umount lustre-OST0000 complete [ 9896.418731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9896.431310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.459300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.048370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.056851] Lustre: Skipped 2 previous similar messages [ 9907.169073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9907.172480] Lustre: Skipped 1 previous similar message [ 9910.240430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9910.819731] Lustre: server umount lustre-MDT0000 complete [ 9915.317336] LustreError: 316778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956885 with bad export cookie 6082667918096242779 [ 9915.328548] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.332319] LustreError: 316778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9915.926395] Lustre: server umount lustre-MDT0001 complete [ 9931.284858] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 9935.970936] Key type lgssc unregistered [ 9936.506952] LNet: 320364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9936.530552] LNetError: 320364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9936.568919] LNet: Removed LNI 192.168.201.130@tcp [ 9937.828801] Key type .llcrypt unregistered [ 9937.832653] Key type ._llcrypt unregistered [ 9963.143251] Key type ._llcrypt registered [ 9963.152596] Key type .llcrypt registered [ 9963.252994] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9964.586086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9964.873167] alg: No test for adler32 (adler32-zlib) [ 9966.008118] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [ 9966.276104] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9967.991590] Key type lgssc registered [ 9969.638762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9978.536460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9978.559653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.069819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.953684] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9989.612868] LustreError: 321785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.644864] LustreError: 321785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9993.580649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.922805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.434837] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10005.737777] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10010.520222] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10019.310743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10019.691060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10026.286359] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10033.957828] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.511956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.525363] Lustre: Skipped 1 previous similar message [10049.529075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.537433] Lustre: Skipped 1 previous similar message [10051.337440] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [10054.625138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.640592] Lustre: Skipped 2 previous similar messages [10055.129261] Lustre: server umount lustre-MDT0000 complete [10056.443488] LustreError: 321780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.416912] LustreError: 322519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957030 with bad export cookie 13445534495767993108 [10059.420405] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10059.428482] LustreError: 322519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10059.746086] LustreError: 321779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.747482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10059.786994] Lustre: Skipped 1 previous similar message [10059.795833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10064.865407] LustreError: 321779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.865425] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10064.882589] LustreError: 321779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10064.898166] Lustre: Skipped 1 previous similar message [10065.878293] Lustre: server umount lustre-MDT0001 complete [10072.704320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10081.045712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10085.361874] LustreError: 321093:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004b4894ff x1873760156161536/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10085.675711] LustreError: 323619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10085.798119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10085.902510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10090.941171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10095.012876] LustreError: 323619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.025724] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10095.042410] LustreError: 323619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10099.148623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10099.604576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10103.862186] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10104.815978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10111.269111] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10112.552336] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10117.859568] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10119.483835] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10125.288062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.290522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10125.300760] Lustre: Skipped 1 previous similar message [10125.320146] Lustre: Skipped 1 previous similar message [10130.456982] Lustre: server umount lustre-OST0000 complete [10135.520877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10135.530769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.553722] Lustre: Skipped 1 previous similar message [10140.590408] Lustre: server umount lustre-MDT0000 complete [10140.641718] LustreError: 325398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10140.653854] LustreError: 325398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10144.796073] LustreError: 321765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957115 with bad export cookie 13445534495768000598 [10144.796893] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10144.807759] LustreError: 321765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10145.182125] Lustre: server umount lustre-MDT0001 complete [10157.390436] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [10160.868989] Key type lgssc unregistered [10161.300473] LNet: 328077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10161.310192] LNetError: 328077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10161.354598] LNet: Removed LNI 192.168.201.130@tcp [10162.205369] Key type .llcrypt unregistered [10162.209961] Key type ._llcrypt unregistered [10188.325667] Key type ._llcrypt registered [10188.327048] Key type .llcrypt registered [10188.440900] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10189.344231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10189.395523] alg: No test for adler32 (adler32-zlib) [10190.591811] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [10190.855778] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [10192.504689] Key type lgssc registered [10194.211780] Lustre: Echo OBD driver; http://www.lustre.org/ [10205.019604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10205.045801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.864865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10212.537377] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10216.423025] LustreError: 329493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.467720] LustreError: 329493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10221.558854] LustreError: 329494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.678243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.274516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10229.945923] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10240.807409] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10246.708396] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10254.312760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.315970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10254.327763] Lustre: Skipped 1 previous similar message [10254.337467] Lustre: Skipped 1 previous similar message [10259.437967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10259.465995] Lustre: Skipped 1 previous similar message [10264.550893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.055181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10265.444967] Lustre: server umount lustre-MDT0000 complete [10268.643378] LustreError: 329489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10268.673290] LustreError: 329489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10271.184572] LustreError: 329476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957241 with bad export cookie 17640797467178955752 [10271.188793] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10271.192793] LustreError: 329476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10271.440360] Lustre: server umount lustre-MDT0001 complete [10286.319351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.549288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10286.579769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10286.790476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10291.325526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10300.705470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.786831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10300.820639] Lustre: 332085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10300.978667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10300.981337] Lustre: Skipped 1 previous similar message [10301.083038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10305.354541] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10313.270869] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10320.406582] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10333.570323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10333.869847] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10333.896764] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10333.906247] LustreError: 333982:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10333.912451] LustreError: 333982:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10333.917487] LustreError: 333982:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10333.972753] Lustre: server umount lustre-OST0000 complete [10333.975478] LustreError: 333982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10346.121782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10346.429492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10346.708680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10354.777068] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10355.426869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10362.321946] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10370.133047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.232665] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10370.321329] Lustre: lustre-OST0001: new disk, initializing [10370.331348] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10370.337958] Lustre: 335525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10370.488115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10376.843909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10376.862288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10377.048190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10377.683835] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10386.101293] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10392.545794] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.562707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10392.563528] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10392.567679] Lustre: Skipped 1 previous similar message [10402.784532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10402.801676] Lustre: Skipped 3 previous similar messages [10404.319254] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10404.542643] Lustre: server umount lustre-OST0001 complete [10407.918427] LustreError: 334450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.936145] LustreError: 334450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10412.000389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10412.005901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.016553] LustreError: Skipped 1 previous similar message [10412.049669] Lustre: Skipped 2 previous similar messages [10413.035806] LustreError: 334449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.062689] LustreError: 334449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10414.476385] Lustre: server umount lustre-OST0000 complete [10419.171667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10419.187060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.210916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.224328] Lustre: Skipped 4 previous similar messages [10423.269340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.901526] Lustre: server umount lustre-MDT0000 complete [10428.235541] LustreError: 334464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957398 with bad export cookie 17640797467178956305 [10428.245952] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10428.246708] LustreError: 334464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10428.713392] Lustre: server umount lustre-MDT0001 complete [10440.459398] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [10443.015222] Key type lgssc unregistered [10443.239737] LNet: 337491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10443.249148] LNetError: 337491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10443.269370] LNet: Removed LNI 192.168.201.130@tcp [10444.196798] Key type .llcrypt unregistered [10444.199414] Key type ._llcrypt unregistered [10518.514452] Key type ._llcrypt registered [10518.516826] Key type .llcrypt registered [10518.595764] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [10536.252945] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10538.169895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10538.266618] alg: No test for adler32 (adler32-zlib) [10539.364205] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [10539.583519] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [10541.279263] Key type lgssc registered [10542.498910] Lustre: Echo OBD driver; http://www.lustre.org/ [10584.266139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10584.283521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.606744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10585.651298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10585.739308] Lustre: lustre-MDT0000: new disk, initializing [10585.876625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.898315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10591.263131] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10602.666847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.820546] Lustre: 342426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10602.871405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10602.875952] Lustre: Skipped 1 previous similar message [10602.946175] Lustre: lustre-MDT0001: new disk, initializing [10603.042177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.089968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10603.103639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10607.020406] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10617.707914] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10623.481878] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10629.797557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10630.002612] Lustre: lustre-OST0000: new disk, initializing [10630.007783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10630.012816] Lustre: 344395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10630.082608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10635.755341] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10640.653334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10640.678920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10640.886555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10647.147980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10654.027177] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10654.328726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10659.196689] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10659.494510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10661.345306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10661.351968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10661.365608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10661.373423] Lustre: Skipped 1 previous similar message [10666.470123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10666.473627] Lustre: Skipped 1 previous similar message [10667.437084] Lustre: server umount lustre-OST0000 complete [10671.587307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10671.596374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.601128] Lustre: Skipped 2 previous similar messages [10671.617643] Lustre: Skipped 1 previous similar message [10676.709496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.715052] Lustre: Skipped 1 previous similar message [10677.840857] Lustre: server umount lustre-MDT0000 complete [10682.629134] LustreError: 342419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957653 with bad export cookie 3178261340728448274 [10682.631128] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10682.635810] LustreError: 342419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10683.024338] Lustre: server umount lustre-MDT0001 complete [10692.253827] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:07:42 (1786957662) [10700.796687] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:07:50 (1786957670) [10710.939948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10711.324183] LustreError: 347153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.334303] LustreError: 347153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10711.399381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10716.199151] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10716.642488] LustreError: 347154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10721.770157] LustreError: 347153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.200868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.568088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10730.152207] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10737.880084] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10744.067958] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10751.366255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.722665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10758.331347] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10765.820944] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10778.842726] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10780.852673] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10782.903783] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10785.092963] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10787.676315] Lustre: Failing over lustre-MDT0000 [10788.035380] Lustre: server umount lustre-MDT0000 complete [10789.156892] LustreError: 347149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.906156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.921371] Lustre: Skipped 1 previous similar message [10794.224715] LustreError: 349272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10794.250972] LustreError: 349272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10804.462683] LustreError: 347149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10804.490155] LustreError: 347149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10808.291256] Lustre: 340648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786957762/real 1786957762] req@0000000051a62c69 x1873760758077184/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786957778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10808.334730] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10808.961151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.537988] LustreError: 340647:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee9f6e98 x1873760758082944/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10818.924432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10819.007269] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10819.815641] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10824.179885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10824.231491] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10824.293351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10824.332915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10835.921131] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10838.289972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10842.384697] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10844.822060] Lustre: Failing over lustre-MDT0000 [10845.070338] Lustre: server umount lustre-MDT0000 complete [10845.451796] LustreError: 347148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10845.462962] LustreError: 347148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10849.764038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.771906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10849.779450] Lustre: Skipped 1 previous similar message [10849.795942] LustreError: Skipped 1 previous similar message [10866.054762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.205889] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10866.207180] Lustre: 340651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786957820/real 1786957820] req@000000005698c560 x1873760758099840/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786957836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10866.565384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10866.651261] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10871.045702] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10871.799159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10871.817363] Lustre: Skipped 2 previous similar messages [10871.866600] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10871.916215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10873.089232] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10883.956882] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10886.178732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10897.379144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.380845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10897.387439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.396950] Lustre: Skipped 3 previous similar messages [10899.180763] Lustre: server umount lustre-MDT0000 complete [10902.498423] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.526648] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10904.239601] LustreError: 347135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957874 with bad export cookie 3178261340728450983 [10904.258001] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.262927] LustreError: 347135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10904.740199] Lustre: server umount lustre-MDT0001 complete [10919.358950] Lustre: server umount lustre-OST0000 complete [10934.791695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10935.407488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10939.870266] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10947.697687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.718951] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10959.219945] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10964.511680] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10971.193386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10977.319347] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10980.704653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10985.211331] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10991.077862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.100750] Lustre: Skipped 1 previous similar message [10991.111903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10991.117847] Lustre: Skipped 2 previous similar messages [10996.411635] Lustre: server umount lustre-OST0000 complete [11006.991362] Lustre: server umount lustre-MDT0000 complete [11010.584737] LustreError: 356298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957981 with bad export cookie 3178261340728451732 [11010.598991] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11010.612409] LustreError: 356298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11011.041455] Lustre: server umount lustre-MDT0001 complete [11021.270845] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11024.087859] Key type lgssc unregistered [11024.355872] LNet: 357051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11024.361868] LNetError: 357051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11024.370741] LNet: Removed LNI 192.168.201.130@tcp [11025.123192] Key type .llcrypt unregistered [11025.126631] Key type ._llcrypt unregistered [11050.174951] Key type ._llcrypt registered [11050.181524] Key type .llcrypt registered [11050.318344] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11051.166529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11051.292398] alg: No test for adler32 (adler32-zlib) [11052.424200] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [11052.675388] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11054.431192] Key type lgssc registered [11055.802845] Lustre: Echo OBD driver; http://www.lustre.org/ [11063.771778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11063.796780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.383023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11070.309835] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11075.555469] LustreError: 358569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.569727] LustreError: 358569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11079.743721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.169553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11084.907158] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11091.877959] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11096.661114] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.794252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.171538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11109.961202] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11111.381944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11116.612128] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11126.755083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11126.785741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11131.360300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11131.372873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11131.390170] Lustre: Skipped 1 previous similar message [11131.403680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11131.413376] Lustre: Skipped 1 previous similar message [11133.010433] Lustre: server umount lustre-MDT0000 complete [11136.760567] LustreError: 358550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958107 with bad export cookie 3266564425029551189 [11136.776716] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11136.789803] LustreError: 358550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11137.090921] Lustre: server umount lustre-MDT0001 complete [11151.566630] Lustre: server umount lustre-OST0000 complete [11165.355863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.998042] LustreError: 362279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.103130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11171.128592] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11171.296839] LustreError: 362280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.332918] LustreError: 362280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11175.395339] LustreError: 362300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.239089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.519569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11180.527326] Lustre: Skipped 2 previous similar messages [11180.686202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11185.861730] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11194.577300] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11200.582510] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11207.952159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.280853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.363688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11214.697419] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11223.027542] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11236.336980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11236.341421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.351446] Lustre: Skipped 1 previous similar message [11240.007447] Lustre: server umount lustre-MDT0000 complete [11241.443666] LustreError: 362274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.454482] LustreError: 362274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11243.520763] LustreError: 362260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958214 with bad export cookie 3266564425029552295 [11243.527394] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11243.534717] LustreError: 362260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11243.965373] Lustre: server umount lustre-MDT0001 complete [11258.387732] Lustre: server umount lustre-OST0000 complete [11276.128801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.799940] LustreError: 366194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.883107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.425783] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11294.268415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.031517] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11310.190144] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11317.323897] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11325.017539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11325.247266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11325.250642] Lustre: Skipped 1 previous similar message [11332.225743] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11340.454505] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11346.739070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11357.152610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11357.155309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11357.160140] Lustre: Skipped 1 previous similar message [11357.163908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11357.167682] Lustre: Skipped 2 previous similar messages [11363.186345] Lustre: server umount lustre-OST0000 complete [11367.417567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11367.422902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.442757] Lustre: Skipped 2 previous similar messages [11367.451906] Lustre: Skipped 2 previous similar messages [11373.687631] Lustre: server umount lustre-MDT0000 complete [11377.212575] LustreError: 366176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958347 with bad export cookie 3266564425029553324 [11377.221180] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11377.226435] LustreError: 366176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11377.780449] Lustre: server umount lustre-MDT0001 complete [11389.857380] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11392.999988] Key type lgssc unregistered [11393.327391] LNet: 369753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11393.348313] LNetError: 369753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11394.429468] LNet: Removed LNI 192.168.201.130@tcp [11395.444298] Key type .llcrypt unregistered [11395.449082] Key type ._llcrypt unregistered [11421.681617] Key type ._llcrypt registered [11421.687803] Key type .llcrypt registered [11421.810713] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11422.854818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11423.036776] alg: No test for adler32 (adler32-zlib) [11424.094522] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [11424.332367] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11426.105386] Key type lgssc registered [11427.459949] Lustre: Echo OBD driver; http://www.lustre.org/ [11435.007123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11435.047829] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.062435] LustreError: 371197:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11435.106544] LustreError: 371197:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11435.117542] LustreError: 371197:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11435.129593] LustreError: 371197:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11435.142486] LustreError: 371197:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11443.000078] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11443.024250] LustreError: 371594:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11443.064516] LustreError: 371594:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11443.082101] LustreError: 371594:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11443.110828] LustreError: 371594:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11443.132518] LustreError: 371594:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11464.100245] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11466.803789] Key type lgssc unregistered [11467.144743] LNet: 372602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11467.153464] LNetError: 372602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11467.174881] LNet: Removed LNI 192.168.201.130@tcp [11468.213542] Key type .llcrypt unregistered [11468.216963] Key type ._llcrypt unregistered [11491.618256] Key type ._llcrypt registered [11491.620935] Key type .llcrypt registered [11491.750503] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [11509.523689] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11511.208129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11511.279897] alg: No test for adler32 (adler32-zlib) [11512.270884] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [11512.472712] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11514.135172] Key type lgssc registered [11515.451460] Lustre: Echo OBD driver; http://www.lustre.org/ [11552.945810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11552.993929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.249347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11554.269387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11554.345527] Lustre: lustre-MDT0000: new disk, initializing [11554.412044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.436781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11558.975283] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11570.504890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.610545] Lustre: 375767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11570.658711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11570.663632] Lustre: Skipped 1 previous similar message [11570.741690] Lustre: lustre-MDT0001: new disk, initializing [11570.833978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11570.866138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11570.883569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11576.094689] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11589.247507] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.786090] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11608.921467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11609.338293] Lustre: lustre-OST0000: new disk, initializing [11609.342274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11609.349234] Lustre: 377738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11609.511616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11615.336592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11615.356690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11615.523043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11615.740983] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11626.486268] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11630.781744] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11631.003137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11636.280695] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11636.582793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11641.322788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.325657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11641.333162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.346731] Lustre: Skipped 1 previous similar message [11644.732423] Lustre: server umount lustre-OST0000 complete [11651.553824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.558157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.583125] Lustre: Skipped 1 previous similar message [11651.608174] Lustre: Skipped 2 previous similar messages [11654.870383] Lustre: server umount lustre-MDT0000 complete [11658.103193] LustreError: 375758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958628 with bad export cookie 8254278083251938410 [11658.121468] LustreError: 375758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11658.134961] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11658.462404] Lustre: server umount lustre-MDT0001 complete [11665.808618] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11667.250305] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:23:57 (1786958637) [11674.789106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.245569] LustreError: 380197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.268294] LustreError: 380197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11675.353497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11679.665725] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11680.739335] LustreError: 380198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.839061] LustreError: 380197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.715729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.001666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11694.935753] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11703.936913] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11709.904907] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11716.204544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11716.458143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11722.233417] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11729.248361] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11735.407251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.524591] Lustre: lustre-OST0001: new disk, initializing [11735.527554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11735.533599] Lustre: 383033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11735.610874] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11740.787489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11740.803820] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11740.872673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11740.987662] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11750.541216] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11756.520198] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11756.527075] LustreError: Skipped 1 previous similar message [11756.528910] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.531804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11756.552493] Lustre: Skipped 1 previous similar message [11760.874622] Lustre: server umount lustre-OST0001 complete [11761.644758] LustreError: 382018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11761.658921] LustreError: 382018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11766.764945] LustreError: 383382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.794688] LustreError: 383382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11777.007744] LustreError: 382015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.021053] LustreError: 382015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11780.065581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11780.079679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11780.081263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11780.095866] Lustre: Skipped 3 previous similar messages [11783.821718] Lustre: server umount lustre-OST0000 complete [11790.327980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11790.332922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11790.340878] Lustre: Skipped 2 previous similar messages [11790.358398] Lustre: Skipped 1 previous similar message [11794.029576] Lustre: server umount lustre-MDT0000 complete [11795.425810] LustreError: 382036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11795.454860] LustreError: 382036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11797.844723] LustreError: 380177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958768 with bad export cookie 8254278083251939299 [11797.871040] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11797.876701] LustreError: 380177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11812.319172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11812.559649] Lustre: server umount lustre-MDT0001 complete [11823.374505] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11826.196848] Key type lgssc unregistered [11826.457532] LNet: 385025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11826.465957] LNetError: 385025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11826.478666] LNet: Removed LNI 192.168.201.130@tcp [11827.350433] Key type .llcrypt unregistered [11827.352418] Key type ._llcrypt unregistered [11877.667333] Key type ._llcrypt registered [11877.672150] Key type .llcrypt registered [11877.852611] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11879.548369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11879.580488] alg: No test for adler32 (adler32-zlib) [11880.804996] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [11881.132627] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11882.839254] Key type lgssc registered [11883.912637] Lustre: Echo OBD driver; http://www.lustre.org/ [11891.597767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11891.623090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11892.924269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11892.949513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11893.152927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11898.207658] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11906.725401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.896579] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11906.948730] Lustre: 387827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11907.265203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11911.972934] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11919.452684] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11925.294499] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11928.037962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11928.047913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.076559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11929.568917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11929.569972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.868622] Lustre: server umount lustre-MDT0000 complete [11937.533293] LustreError: 387821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958908 with bad export cookie 6421768373305968966 [11937.540193] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.548112] LustreError: 387821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11937.947915] Lustre: server umount lustre-MDT0001 complete [11948.537311] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:28:37 (1786958917) [11955.119371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.968785] LustreError: 390340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.994760] LustreError: 390340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11956.075090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11961.318442] LustreError: 390341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.335536] LustreError: 390341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11961.773176] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11966.443442] LustreError: 390340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.798478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.236342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11976.107873] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11983.319192] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11989.338544] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11996.851982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11997.078961] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11997.272126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12004.255759] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12012.877972] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12020.400461] Lustre: Permanently deactivating lustre-OST0000 [12020.423331] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12027.430726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12029.735866] LustreError: 393325:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12029.742113] LustreError: 393325:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12029.753524] LustreError: 393325:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12032.312478] LustreError: 393374:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12032.318165] LustreError: 393374:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12040.239832] Lustre: server umount lustre-OST0000 complete [12043.056331] LustreError: 393624:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12043.059909] LustreError: 393624:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12045.280464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12045.292351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12045.294827] Lustre: Skipped 1 previous similar message [12045.309072] Lustre: Skipped 2 previous similar messages [12050.400117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12050.405822] Lustre: Skipped 1 previous similar message [12054.596458] Lustre: server umount lustre-MDT0000 complete [12055.521284] LustreError: 390337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12055.540412] LustreError: 390337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12058.211414] LustreError: 390321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959028 with bad export cookie 6421768373305969547 [12058.215535] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12058.222893] LustreError: 390321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12068.002707] Lustre: server umount lustre-MDT0001 complete [12078.993583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.635212] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12087.174863] LustreError: 394783:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12093.177929] Lustre: 395122:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.130@tcp [12095.518591] Lustre: server umount MGS complete [12106.853331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12107.236288] LustreError: 395846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.255737] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12107.263547] Lustre: Skipped 1 previous similar message [12107.318133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12111.776360] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12117.508329] LustreError: 395847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.534648] LustreError: 395847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12120.820451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.127977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12125.711784] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12131.992281] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12137.168393] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12139.051393] Lustre: Permanently reactivating lustre-OST0000 [12139.063770] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12139.069486] Lustre: Skipped 2 previous similar messages [12165.087118] LustreError: 397581:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12171.104647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12171.392185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12171.423769] Lustre: Skipped 1 previous similar message [12177.121181] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12184.211799] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12190.176205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12190.177321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12190.183526] LustreError: Skipped 1 previous similar message [12190.184870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12190.203603] Lustre: Skipped 1 previous similar message [12190.219411] Lustre: Skipped 1 previous similar message [12195.778157] Lustre: server umount lustre-OST0000 complete [12200.417267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.444162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12200.446534] Lustre: Skipped 1 previous similar message [12200.471530] Lustre: Skipped 3 previous similar messages [12206.633920] Lustre: server umount lustre-MDT0000 complete [12210.140980] LustreError: 395828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959180 with bad export cookie 6421768373305970597 [12210.146240] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12210.152255] LustreError: 395828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12210.709382] Lustre: server umount lustre-MDT0001 complete [12222.596592] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12225.522533] Key type lgssc unregistered [12225.950585] LNet: 400215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12225.955341] LNetError: 400215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12225.978660] LNet: Removed LNI 192.168.201.130@tcp [12226.938859] Key type .llcrypt unregistered [12226.942195] Key type ._llcrypt unregistered [12245.350209] Key type ._llcrypt registered [12245.354813] Key type .llcrypt registered [12245.451510] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [12260.753339] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12261.812956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12261.945512] alg: No test for adler32 (adler32-zlib) [12263.084046] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [12263.401762] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12265.151175] Key type lgssc registered [12266.337916] Lustre: Echo OBD driver; http://www.lustre.org/ [12302.774626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12302.834113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.127371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12304.157223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12304.251508] Lustre: lustre-MDT0000: new disk, initializing [12304.341403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.386610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12308.666540] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12320.879015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12321.072425] Lustre: 403369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12321.134557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12321.142663] Lustre: Skipped 1 previous similar message [12321.236181] Lustre: lustre-MDT0001: new disk, initializing [12321.372957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12321.435310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12321.453858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12326.638240] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12336.448579] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12342.253691] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12349.247265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12349.691667] Lustre: lustre-OST0000: new disk, initializing [12349.694602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12349.708398] Lustre: 405339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12349.846674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12356.706657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12356.715824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12356.908318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12357.291939] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12367.282550] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12372.457140] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12372.704222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12377.885688] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12378.174868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12382.689540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12382.707622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.714693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12382.727562] Lustre: Skipped 1 previous similar message [12382.755093] Lustre: Skipped 1 previous similar message [12386.537813] Lustre: server umount lustre-OST0000 complete [12392.929637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12392.932904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12392.953861] LustreError: Skipped 1 previous similar message [12392.964103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12396.509469] Lustre: server umount lustre-MDT0000 complete [12400.513481] LustreError: 404166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959371 with bad export cookie 5717215175163437901 [12400.514797] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12400.543883] LustreError: 404166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12401.011473] Lustre: server umount lustre-MDT0001 complete [12409.913370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12412.100805] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:36:21 (1786959381) [12419.854135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.421683] LustreError: 407800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.436213] LustreError: 407800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12420.496074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12424.832835] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12425.724695] LustreError: 407801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.820777] LustreError: 407800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.216705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.556753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12439.453224] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12446.789834] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12453.797874] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12461.097657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12461.486478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12467.537416] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12475.609968] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12478.165824] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12481.781521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12489.187420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12489.197612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12489.214671] Lustre: Skipped 2 previous similar messages [12489.229318] Lustre: Skipped 1 previous similar message [12493.957677] Lustre: server umount lustre-OST0000 complete [12499.429114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12499.443231] Lustre: Skipped 1 previous similar message [12499.457057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12499.462965] Lustre: Skipped 1 previous similar message [12504.545035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.552443] Lustre: Skipped 1 previous similar message [12512.226558] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12512.828975] Lustre: server umount lustre-MDT0000 complete [12513.761974] LustreError: 407796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12513.792413] LustreError: 407796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12517.947846] LustreError: 407780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959488 with bad export cookie 5717215175163438783 [12517.953506] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12517.968291] LustreError: 407780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12518.686842] Lustre: server umount lustre-MDT0001 complete [12534.786506] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12538.309496] Key type lgssc unregistered [12538.526085] LNet: 411532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12538.534044] LNetError: 411532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12538.547322] LNet: Removed LNI 192.168.201.130@tcp [12539.482687] Key type .llcrypt unregistered [12539.487814] Key type ._llcrypt unregistered [12565.125350] Key type ._llcrypt registered [12565.128153] Key type .llcrypt registered [12565.318709] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12565.719601] Key type .llcrypt unregistered [12565.720948] Key type ._llcrypt unregistered [12583.995869] Key type ._llcrypt registered [12583.997779] Key type .llcrypt registered [12584.101472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12585.073511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12585.228087] alg: No test for adler32 (adler32-zlib) [12586.420567] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [12586.705554] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12588.455954] Key type lgssc registered [12589.508695] Lustre: Echo OBD driver; http://www.lustre.org/ [12598.606624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12598.620673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.207986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12605.510741] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12609.509257] LustreError: 413750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12609.554791] LustreError: 413750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12614.626577] LustreError: 413751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.100887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12615.445501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12620.774848] LustreError: 413747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.801230] LustreError: 413747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12622.495855] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12624.864686] LustreError: 413750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.887661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12624.893141] LustreError: 413750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12629.990939] LustreError: 413751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.011262] LustreError: 413751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12630.722258] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12638.478056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.093685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12644.134074] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12654.536067] Lustre: server umount lustre-OST0000 complete [12659.682165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.686224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12659.709807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12664.800486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12664.815315] Lustre: Skipped 2 previous similar messages [12669.026262] Lustre: server umount lustre-MDT0000 complete [12669.921230] LustreError: 414493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.934248] LustreError: 414493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12670.964939] Lustre: 413060:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786959625/real 1786959625] req@000000009f6ecd83 x1873762904468992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786959641 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12670.966827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.000554] Lustre: 413060:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12671.025228] Lustre: Skipped 1 previous similar message [12673.408093] LustreError: 415857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959644 with bad export cookie 805314139034109961 [12673.419259] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12673.424975] LustreError: 415857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12673.903868] Lustre: server umount lustre-MDT0001 complete [12687.202789] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12691.015854] Key type lgssc unregistered [12691.413956] LNet: 416899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.422309] LNetError: 416899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.444257] LNet: Removed LNI 192.168.201.130@tcp [12692.432391] Key type .llcrypt unregistered [12692.434558] Key type ._llcrypt unregistered [12723.352886] Key type ._llcrypt registered [12723.356848] Key type .llcrypt registered [12723.503219] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12725.241590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12725.257936] alg: No test for adler32 (adler32-zlib) [12726.556278] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [12726.886546] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12728.575297] Key type lgssc registered [12729.914186] Lustre: Echo OBD driver; http://www.lustre.org/ [12738.917390] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12738.951550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12754.527748] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12759.647391] LustreError: 418293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [12786.399395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12793.954075] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12802.831338] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12810.043415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12810.520589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12810.595089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12815.770782] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12815.844303] LustreError: 418318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.863490] LustreError: 418318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12820.976103] LustreError: 418319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12821.017282] LustreError: 418319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12826.090486] LustreError: 418318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12826.122475] LustreError: 418318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12826.314590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.756334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12831.956679] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12842.073778] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12848.562772] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12856.805458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12856.813872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.829568] LustreError: 418310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.950576] Lustre: server umount lustre-OST0000 complete [12861.428178] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12861.440994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.455821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.948217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.957684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.661564] Lustre: server umount lustre-MDT0000 complete [12868.067189] LustreError: 419336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12868.090426] LustreError: 419336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12871.242243] LustreError: 418301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959841 with bad export cookie 10476955801061765471 [12871.254983] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12871.258443] LustreError: 418301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12881.137686] Lustre: server umount lustre-MDT0001 complete [12893.121750] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12896.748245] Key type lgssc unregistered [12897.029127] LNet: 421869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12897.038773] LNetError: 421869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12897.060711] LNet: Removed LNI 192.168.201.130@tcp [12898.022376] Key type .llcrypt unregistered [12898.024093] Key type ._llcrypt unregistered [12926.825854] Key type ._llcrypt registered [12926.831173] Key type .llcrypt registered [12927.012520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12929.101904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12929.183177] alg: No test for adler32 (adler32-zlib) [12930.483813] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [12931.011214] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12932.848411] Key type lgssc registered [12934.164784] Lustre: Echo OBD driver; http://www.lustre.org/ [12944.687306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12944.728177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12946.572974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12951.039812] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12957.184729] LustreError: 423286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12957.211214] LustreError: 423286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12961.067613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12961.480157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12966.201146] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12975.802971] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12982.736832] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12990.381804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12990.979988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12991.987496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12998.929526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13008.982533] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13015.882385] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [13017.570118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13017.579761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13017.593687] Lustre: Skipped 1 previous similar message [13017.610637] Lustre: Skipped 2 previous similar messages [13019.341953] Lustre: server umount lustre-MDT0000 complete [13020.922898] LustreError: 423283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13022.411425] LustreError: 423267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.30@tcp arrived at 1786959993 with bad export cookie 893824145887721064 [13022.424195] LustreError: 423267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13022.689711] LustreError: 425166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13022.718984] LustreError: 425166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13024.334704] LustreError: 423269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959995 with bad export cookie 893824145887720168 [13024.350467] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13027.810078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13027.811152] LustreError: 423283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13027.816830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13027.816842] Lustre: Skipped 1 previous similar message [13027.818788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13027.841803] LustreError: 423283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13029.859573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13030.754298] Lustre: server umount lustre-OST0000 complete [13034.977305] LustreError: 423283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.989051] LustreError: 423283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13038.995603] Lustre: server umount lustre-MDT0001 complete [13052.205957] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13056.173812] Key type lgssc unregistered [13056.695801] LNet: 426974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13056.705456] LNetError: 426974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13056.725529] LNet: Removed LNI 192.168.201.130@tcp [13057.737305] Key type .llcrypt unregistered [13057.740161] Key type ._llcrypt unregistered [13087.179619] Key type ._llcrypt registered [13087.182969] Key type .llcrypt registered [13087.302714] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13088.733175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13088.746125] alg: No test for adler32 (adler32-zlib) [13089.868653] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [13090.030202] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13091.664813] Key type lgssc registered [13093.282804] Lustre: Echo OBD driver; http://www.lustre.org/ [13102.754479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13102.786353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.392499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13109.523150] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13114.849582] LustreError: 428394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13114.864702] LustreError: 428394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13119.969583] LustreError: 428393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.404453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13120.830292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13126.599664] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13136.197248] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13143.898980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13151.950201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13152.401966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13159.836964] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13169.149813] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13174.986591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13180.403278] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.424928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13180.430080] Lustre: Skipped 1 previous similar message [13181.192670] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13184.938291] Lustre: server umount lustre-MDT0001 complete [13185.513440] LustreError: 428393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.555579] LustreError: 428393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13190.649568] LustreError: 428388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.668752] LustreError: 428388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13195.747726] LustreError: 428394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13195.749804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13195.763486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13195.807946] Lustre: Skipped 1 previous similar message [13195.815745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13199.503818] Lustre: server umount lustre-OST0000 complete [13205.726229] Lustre: server umount lustre-MDT0000 complete [13223.514494] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13226.377727] Key type lgssc unregistered [13226.785066] LNet: 432083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13226.809611] LNetError: 432083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13226.836702] LNet: Removed LNI 192.168.201.130@tcp [13228.014195] Key type .llcrypt unregistered [13228.022142] Key type ._llcrypt unregistered [13258.572785] Key type ._llcrypt registered [13258.575254] Key type .llcrypt registered [13258.674390] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [13275.780972] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13277.456661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13277.554790] alg: No test for adler32 (adler32-zlib) [13278.649672] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [13279.007488] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13280.815222] Key type lgssc registered [13282.506462] Lustre: Echo OBD driver; http://www.lustre.org/ [13326.819392] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13346.535213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13346.564794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13347.941666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13347.975236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13348.075090] Lustre: lustre-MDT0000: new disk, initializing [13348.175300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13348.220146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13353.371883] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13374.204954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13374.654687] Lustre: lustre-OST0000: new disk, initializing [13374.660996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13374.664117] Lustre: Skipped 1 previous similar message [13374.669598] Lustre: 437121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13374.792621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13381.315101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13381.332144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13381.381888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13384.188523] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13395.081831] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13414.969348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13415.158467] Lustre: 436103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13415.246245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13415.437683] Lustre: lustre-MDT0001: new disk, initializing [13415.636328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13415.735907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13415.749029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13421.688856] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13432.423717] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13432.689473] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13437.440966] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13437.659567] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13443.846173] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13444.578776] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.606844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13444.631866] Lustre: Skipped 1 previous similar message [13448.891190] Lustre: server umount lustre-MDT0001 complete [13454.067751] LustreError: 436111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.087865] LustreError: 436111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13454.824743] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.869372] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13459.952581] LustreError: 436109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.979563] LustreError: 436109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13465.062922] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13465.099280] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13470.178346] LustreError: 436114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13470.180908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.240867] Lustre: Skipped 1 previous similar message [13470.247785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13475.300001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13475.796246] Lustre: server umount lustre-OST0000 complete [13480.430279] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.440024] LustreError: 436115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13486.955251] Lustre: server umount lustre-MDT0000 complete [13503.949715] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13507.511535] Key type lgssc unregistered [13507.850787] LNet: 440876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13507.861612] LNetError: 440876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13507.895381] LNet: Removed LNI 192.168.201.130@tcp [13508.952498] Key type .llcrypt unregistered [13508.960394] Key type ._llcrypt unregistered [13555.386283] Key type ._llcrypt registered [13555.389222] Key type .llcrypt registered [13555.525131] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13556.013419] Key type .llcrypt unregistered [13556.015103] Key type ._llcrypt unregistered [13573.900799] Key type ._llcrypt registered [13573.903891] Key type .llcrypt registered [13574.058458] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13575.567141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13575.753229] alg: No test for adler32 (adler32-zlib) [13577.087954] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [13577.408299] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13579.159714] Key type lgssc registered [13580.860075] Lustre: Echo OBD driver; http://www.lustre.org/ [13611.427375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13611.469367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13612.671281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13612.698663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13612.772272] Lustre: lustre-MDT0000: new disk, initializing [13612.935925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13612.955566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13617.013226] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13628.485384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13628.646282] Lustre: 444263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13628.718825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13628.726901] Lustre: Skipped 1 previous similar message [13628.822257] Lustre: lustre-MDT0001: new disk, initializing [13628.970384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13629.036606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13629.065700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13634.735553] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13645.083227] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13651.415558] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13660.192748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13660.669615] Lustre: lustre-OST0000: new disk, initializing [13660.684070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13660.695898] Lustre: 446231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13660.847078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13669.057486] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13670.014763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13670.020816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13670.146753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13680.037619] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13690.336562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13690.352081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13690.373149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13690.857178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13690.882632] Lustre: Skipped 1 previous similar message [13690.892400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13690.894444] Lustre: Skipped 1 previous similar message [13693.464828] Lustre: server umount lustre-MDT0000 complete [13697.905995] LustreError: 444255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960668 with bad export cookie 8013522362876252281 [13697.908279] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13697.924302] LustreError: 444255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13698.595048] Lustre: server umount lustre-MDT0001 complete [13714.202357] Lustre: server umount lustre-OST0000 complete [13727.294693] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:58:17 (1786960697) [13737.177736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.928035] LustreError: 448261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.952105] LustreError: 448261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13738.044944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13742.448222] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13743.074274] LustreError: 448262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13743.091069] LustreError: 448262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13748.200085] LustreError: 448261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13753.328171] LustreError: 448262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13754.948500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.438467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13761.968379] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13770.718245] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13777.494992] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13799.024020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13799.332347] Lustre: Found index 0 for lustre-OST0000, updating log [13799.399751] LustreError: 450087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.644344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13807.707643] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13809.223528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13816.793673] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13824.490807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13824.494477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13824.505703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13831.649604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13831.665971] Lustre: Skipped 4 previous similar messages [13837.807071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13837.830097] Lustre: Skipped 1 previous similar message [13838.306879] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13838.674711] Lustre: server umount lustre-OST0000 complete [13857.764731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13857.780527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13857.796943] Lustre: Skipped 1 previous similar message [13857.808052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13862.491240] Lustre: server umount lustre-MDT0000 complete [13864.422478] LustreError: 448990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13864.466746] LustreError: 448990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13865.247535] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [13866.578942] LustreError: 448243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960837 with bad export cookie 8013522362876253667 [13866.587960] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13866.598152] LustreError: 448243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13867.040138] Lustre: server umount lustre-MDT0001 complete [13880.049344] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:00:49 (1786960849) [14510.287033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14511.380425] LustreError: 454688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14511.572392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14516.708278] LustreError: 454689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14517.964791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14521.826029] LustreError: 454688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14528.101692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14528.722962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14534.512960] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14543.288846] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14549.196361] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14569.456551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14569.655766] Lustre: Found index 0 for lustre-OST0000, updating log [14569.761181] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [14569.937195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14576.838383] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14584.661423] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14587.360210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14587.366603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14587.375939] Lustre: Skipped 1 previous similar message [14587.385645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14587.391085] Lustre: Skipped 2 previous similar messages [14592.866650] Lustre: server umount lustre-OST0000 complete [14597.604942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14597.611326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14597.630584] Lustre: Skipped 1 previous similar message [14601.769079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14601.771212] Lustre: Skipped 7 previous similar messages [14603.056433] Lustre: server umount lustre-MDT0000 complete [14603.825265] LustreError: 455417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14603.847543] LustreError: 455417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14606.726883] LustreError: 454669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786961577 with bad export cookie 8013522362876254759 [14606.731611] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14606.749571] LustreError: 454669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14607.362872] Lustre: server umount lustre-MDT0001 complete [14630.175305] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [14638.508977] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [14650.021873] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14690.204697] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:14:19 (1786961659) [14709.740548] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [14712.719659] Key type lgssc unregistered [14713.100400] LNet: 461307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14713.109343] LNetError: 461307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14832.943271] LNet: Removed LNI 192.168.201.130@tcp [14833.879254] Key type .llcrypt unregistered [14833.882477] Key type ._llcrypt unregistered [14849.959448] Key type ._llcrypt registered [14849.965410] Key type .llcrypt registered [14850.136568] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14851.832068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14852.120216] alg: No test for adler32 (adler32-zlib) [14853.254479] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [14853.668700] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [14855.528175] Key type lgssc registered [14856.787179] Lustre: Echo OBD driver; http://www.lustre.org/ [14861.066762] LNet: Added LNI 192.168.201.130@tcp100 [8/256/0/180] [14861.997795] LNet: Added LNI 192.168.201.130@tcp101 [8/256/0/180] [14862.899544] LNet: Added LNI 192.168.201.130@tcp102 [8/256/0/180] [14863.737703] LNet: Added LNI 192.168.201.130@tcp103 [8/256/0/180] [14864.634692] LNet: Added LNI 192.168.201.130@tcp104 [8/256/0/180] [14865.625438] LNet: Added LNI 192.168.201.130@tcp105 [8/256/0/180] [14866.730806] LNet: Added LNI 192.168.201.130@tcp106 [8/256/0/180] [14867.722879] LNet: Added LNI 192.168.201.130@tcp107 [8/256/0/180] [14868.654875] LNet: Added LNI 192.168.201.130@tcp108 [8/256/0/180] [14869.456443] LNet: Added LNI 192.168.201.130@tcp109 [8/256/0/180] [14870.470485] LNet: Added LNI 192.168.201.130@tcp110 [8/256/0/180] [14871.511747] LNet: Added LNI 192.168.201.130@tcp111 [8/256/0/180] [14872.530717] LNet: Added LNI 192.168.201.130@tcp112 [8/256/0/180] [14873.388490] LNet: Added LNI 192.168.201.130@tcp113 [8/256/0/180] [14874.241392] LNet: Added LNI 192.168.201.130@tcp114 [8/256/0/180] [14875.084174] LNet: Added LNI 192.168.201.130@tcp115 [8/256/0/180] [14875.941481] LNet: Added LNI 192.168.201.130@tcp116 [8/256/0/180] [14876.681806] LNet: Added LNI 192.168.201.130@tcp117 [8/256/0/180] [14877.723866] LNet: Added LNI 192.168.201.130@tcp118 [8/256/0/180] [14879.234867] LNet: Added LNI 192.168.201.130@tcp119 [8/256/0/180] [14880.560183] LNet: Added LNI 192.168.201.130@tcp120 [8/256/0/180] [14881.825627] LNet: Added LNI 192.168.201.130@tcp121 [8/256/0/180] [14883.007567] LNet: Added LNI 192.168.201.130@tcp122 [8/256/0/180] [14884.121357] LNet: Added LNI 192.168.201.130@tcp123 [8/256/0/180] [14885.072551] LNet: Added LNI 192.168.201.130@tcp124 [8/256/0/180] [14886.035072] LNet: Added LNI 192.168.201.130@tcp125 [8/256/0/180] [14886.905240] LNet: Added LNI 192.168.201.130@tcp126 [8/256/0/180] [14887.780295] LNet: Added LNI 192.168.201.130@tcp127 [8/256/0/180] [14888.741868] LNet: Added LNI 192.168.201.130@tcp128 [8/256/0/180] [14890.048087] LNet: Added LNI 192.168.201.130@tcp129 [8/256/0/180] [14891.155741] LNet: Added LNI 192.168.201.130@tcp130 [8/256/0/180] [14892.045690] LNet: Added LNI 192.168.201.130@tcp131 [8/256/0/180] [14893.225356] LNet: Added LNI 192.168.201.130@tcp132 [8/256/0/180] [14894.418520] LNet: Added LNI 192.168.201.130@tcp133 [8/256/0/180] [14895.489498] LNet: Added LNI 192.168.201.130@tcp134 [8/256/0/180] [14896.696041] LNet: Added LNI 192.168.201.130@tcp135 [8/256/0/180] [14905.388289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14905.438733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14906.605509] Lustre: 464344:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [14906.717405] LNetError: 461802:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14906.735831] LNetError: Protocol error connecting to 192.168.201.130@tcp113 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14906.753905] LNetError: Skipped 11 previous similar messages [14906.904647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14906.911184] LNet: 8 local NIs in recovery (showing 8): 192.168.201.130@tcp121, 192.168.201.130@tcp124, 192.168.201.130@tcp125, 192.168.201.130@tcp126, 192.168.201.130@tcp128, 192.168.201.130@tcp129, 192.168.201.130@tcp123, 192.168.201.130@tcp127 [14906.984178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14907.077586] Lustre: lustre-MDT0000: new disk, initializing [14907.170942] Lustre: 464401:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [14907.181612] Lustre: 464401:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14907.207071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14907.248532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14907.810939] LNetError: 461802:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14907.827839] LNetError: 461802:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14907.835821] LNetError: Protocol error connecting to 192.168.201.130@tcp103 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14913.090653] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14913.126062] LNetError: 464365:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14913.141276] LNetError: 464365:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [14913.153933] LNetError: Protocol error connecting to 192.168.201.130@tcp121 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14913.172423] LNetError: Skipped 13 previous similar messages [14921.317503] LNetError: 464370:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14921.336366] LNetError: 464370:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14921.357360] LNetError: Protocol error connecting to 192.168.201.130@tcp121 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14924.972363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14925.153432] Lustre: 464363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14925.237382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14925.241819] Lustre: Skipped 1 previous similar message [14925.334952] Lustre: lustre-MDT0001: new disk, initializing [14925.378539] Lustre: 465162:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [14925.383909] Lustre: 465162:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14925.514376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14925.567710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14925.574392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14931.406984] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14933.565575] Lustre: 465525:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [14933.572570] Lustre: 465525:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14937.702234] LNetError: 464373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14937.719233] LNetError: Protocol error connecting to 192.168.201.130@tcp121 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14941.874606] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14946.913557] LNetError: 461801:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14946.928864] LNetError: 461801:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14946.937695] LNetError: Protocol error connecting to 192.168.201.130@tcp123 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14946.944321] LNetError: Skipped 5 previous similar messages [14947.651623] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14954.604545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14954.861292] Lustre: lustre-OST0000: new disk, initializing [14954.864785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14954.870852] Lustre: 466335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14954.956357] Lustre: 466340:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [14954.962388] Lustre: 466340:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14955.016218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14960.737456] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14960.768962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14960.787168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14960.957761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14963.302701] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [14963.324301] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14963.339685] LNetError: Protocol error connecting to 192.168.201.130@tcp103 at host 192.168.201.130:988: is it running a compatible version of Lustre? [14963.376093] LNetError: Skipped 13 previous similar messages [14966.374500] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp121 [14970.374729] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14981.621220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14981.625536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14981.629234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14981.636223] Lustre: Skipped 1 previous similar message [14981.670593] Lustre: Skipped 1 previous similar message [14986.720741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14986.734310] Lustre: Skipped 1 previous similar message [14991.845223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14991.854523] Lustre: Skipped 1 previous similar message [14992.355508] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14992.722022] Lustre: server umount lustre-OST0000 complete [15002.081134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15002.088276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15002.092713] Lustre: Skipped 1 previous similar message [15002.112973] Lustre: Skipped 1 previous similar message [15003.232807] LNetError: 464377:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15003.245799] LNetError: 464377:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [15003.251343] LNetError: Protocol error connecting to 192.168.201.130@tcp119 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15003.265038] LNetError: Skipped 20 previous similar messages [15004.048364] Lustre: server umount lustre-MDT0000 complete [15007.759144] LustreError: 464354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786961978 with bad export cookie 5655956001631306982 [15007.767733] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15007.773903] LustreError: 464354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15008.160812] Lustre: server umount lustre-MDT0001 complete [15020.366656] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15023.246819] Key type lgssc unregistered [15023.663400] LNet: 468252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15023.671896] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15024.735663] LNet: Removed LNI 192.168.201.130@tcp [15024.745976] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15024.751362] LNet: Removed LNI 192.168.201.130@tcp100 [15025.759231] LNet: Removed LNI 192.168.201.130@tcp101 [15025.763876] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15025.775040] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15025.795388] LNet: Removed LNI 192.168.201.130@tcp102 [15026.847250] LNet: Removed LNI 192.168.201.130@tcp103 [15027.871658] LNet: Removed LNI 192.168.201.130@tcp104 [15027.876146] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15027.885585] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15027.898938] LNet: Removed LNI 192.168.201.130@tcp105 [15028.964220] LNet: Removed LNI 192.168.201.130@tcp106 [15029.983200] LNet: Removed LNI 192.168.201.130@tcp107 [15031.007777] LNet: Removed LNI 192.168.201.130@tcp108 [15032.031333] LNet: Removed LNI 192.168.201.130@tcp109 [15032.033662] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15032.043237] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15032.056083] LNet: Removed LNI 192.168.201.130@tcp110 [15033.120424] LNet: Removed LNI 192.168.201.130@tcp111 [15034.143310] LNet: Removed LNI 192.168.201.130@tcp112 [15035.168085] LNet: Removed LNI 192.168.201.130@tcp113 [15036.191736] LNet: Removed LNI 192.168.201.130@tcp114 [15037.215455] LNet: Removed LNI 192.168.201.130@tcp115 [15038.239324] LNet: Removed LNI 192.168.201.130@tcp116 [15039.263274] LNet: Removed LNI 192.168.201.130@tcp117 [15040.287300] LNet: Removed LNI 192.168.201.130@tcp118 [15040.292461] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15040.299715] LNetError: 468252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15040.309143] LNet: Removed LNI 192.168.201.130@tcp119 [15041.375292] LNet: Removed LNI 192.168.201.130@tcp120 [15041.377440] LNet: Removed LNI 192.168.201.130@tcp121 [15042.399176] LNet: Removed LNI 192.168.201.130@tcp122 [15043.423446] LNet: Removed LNI 192.168.201.130@tcp123 [15044.447381] LNet: Removed LNI 192.168.201.130@tcp124 [15045.471250] LNet: Removed LNI 192.168.201.130@tcp125 [15046.495229] LNet: Removed LNI 192.168.201.130@tcp126 [15047.521050] LNet: Removed LNI 192.168.201.130@tcp127 [15048.545602] LNet: Removed LNI 192.168.201.130@tcp128 [15049.632283] LNet: Removed LNI 192.168.201.130@tcp129 [15049.640675] LNet: Removed LNI 192.168.201.130@tcp130 [15049.661513] LNet: Removed LNI 192.168.201.130@tcp131 [15049.675672] LNet: Removed LNI 192.168.201.130@tcp132 [15049.683846] LNet: Removed LNI 192.168.201.130@tcp133 [15049.691569] LNet: Removed LNI 192.168.201.130@tcp134 [15049.716039] LNet: Removed LNI 192.168.201.130@tcp135 [15050.770917] Key type .llcrypt unregistered [15050.773453] Key type ._llcrypt unregistered [15082.173580] Key type ._llcrypt registered [15082.176461] Key type .llcrypt registered [15082.312993] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [15098.012396] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15099.499251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15099.616937] alg: No test for adler32 (adler32-zlib) [15100.761549] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [15101.081623] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15102.775259] Key type lgssc registered [15104.022524] Lustre: Echo OBD driver; http://www.lustre.org/ [15141.187432] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:21:50 (1786962110) [15160.036329] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15163.199920] Key type lgssc unregistered [15163.532665] LNet: 472343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15163.545358] LNetError: 472343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15163.569697] LNet: Removed LNI 192.168.201.130@tcp [15164.668913] Key type .llcrypt unregistered [15164.670752] Key type ._llcrypt unregistered [15183.776884] Key type ._llcrypt registered [15183.778362] Key type .llcrypt registered [15183.968751] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15185.251044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15185.364655] alg: No test for adler32 (adler32-zlib) [15186.714617] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [15187.231803] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15189.015969] Key type lgssc registered [15190.681983] Lustre: Echo OBD driver; http://www.lustre.org/ [15202.536113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15202.551107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15203.955330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15204.000599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15204.175643] Lustre: lustre-MDT0000: new disk, initializing [15204.269296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15204.283912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15209.669229] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15220.554618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15220.659399] Lustre: 473558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15220.718720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15220.723568] Lustre: Skipped 1 previous similar message [15220.793719] Lustre: lustre-MDT0001: new disk, initializing [15220.891780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15220.942625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15220.971622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15225.790992] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15235.544546] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15241.422481] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15248.728202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15248.965330] Lustre: lustre-OST0000: new disk, initializing [15248.968927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15248.975868] Lustre: 475526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15249.052949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15256.143307] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15256.167379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15256.183947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15256.342515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15267.238394] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15272.271838] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing lnet_if_list [15278.288180] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15278.321544] LNet: Added LNI 192.168.201.130@tcp10 [8/256/0/180] [15284.771428] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15284.794905] LNet: Added LNI 192.168.201.130@tcp11 [8/256/0/180] [15289.510951] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15289.533850] LNet: Added LNI 192.168.201.130@tcp12 [8/256/0/180] [15294.921270] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15294.938254] LNet: Added LNI 192.168.201.130@tcp13 [8/256/0/180] [15299.473575] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15299.548618] LNet: Added LNI 192.168.201.130@tcp14 [8/256/0/180] [15305.436843] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15305.472410] LNet: Added LNI 192.168.201.130@tcp15 [8/256/0/180] [15312.407193] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15312.444942] LNet: Added LNI 192.168.201.130@tcp16 [8/256/0/180] [15318.720472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15318.805311] LNet: Added LNI 192.168.201.130@tcp17 [8/256/0/180] [15325.757596] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15325.818247] LNet: Added LNI 192.168.201.130@tcp18 [8/256/0/180] [15332.334617] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15332.358932] LNet: Added LNI 192.168.201.130@tcp19 [8/256/0/180] [15338.632963] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15338.670496] LNet: Added LNI 192.168.201.130@tcp20 [8/256/0/180] [15345.550550] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15345.605405] LNet: Added LNI 192.168.201.130@tcp21 [8/256/0/180] [15352.336165] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15352.358739] LNet: Added LNI 192.168.201.130@tcp22 [8/256/0/180] [15358.176573] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15358.199650] LNet: Added LNI 192.168.201.130@tcp23 [8/256/0/180] [15363.605438] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15363.645335] LNet: Added LNI 192.168.201.130@tcp24 [8/256/0/180] [15370.315981] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15370.343206] LNet: Added LNI 192.168.201.130@tcp25 [8/256/0/180] [15376.873526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15376.916872] LNet: Added LNI 192.168.201.130@tcp26 [8/256/0/180] [15385.402551] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15385.475425] LNet: Added LNI 192.168.201.130@tcp27 [8/256/0/180] [15393.271723] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15393.304199] LNet: Added LNI 192.168.201.130@tcp28 [8/256/0/180] [15400.826370] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15400.878702] LNet: Added LNI 192.168.201.130@tcp29 [8/256/0/180] [15410.218121] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15410.257397] LNet: Added LNI 192.168.201.130@tcp30 [8/256/0/180] [15419.004056] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15419.039815] LNet: Added LNI 192.168.201.130@tcp31 [8/256/0/180] [15426.301460] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15426.363037] LNet: Added LNI 192.168.201.130@tcp32 [8/256/0/180] [15435.151526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15435.235183] LNet: Added LNI 192.168.201.130@tcp33 [8/256/0/180] [15443.375720] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15443.406278] LNet: Added LNI 192.168.201.130@tcp34 [8/256/0/180] [15451.963426] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15452.042587] LNet: Added LNI 192.168.201.130@tcp35 [8/256/0/180] [15460.149624] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15460.189556] LNet: Added LNI 192.168.201.130@tcp36 [8/256/0/180] [15468.841131] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15468.881877] LNet: Added LNI 192.168.201.130@tcp37 [8/256/0/180] [15477.385971] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15477.411128] LNet: Added LNI 192.168.201.130@tcp38 [8/256/0/180] [15486.620563] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15486.646589] LNet: Added LNI 192.168.201.130@tcp39 [8/256/0/180] [15495.202419] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15495.235479] LNet: Added LNI 192.168.201.130@tcp40 [8/256/0/180] [15504.117852] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15504.165671] LNet: Added LNI 192.168.201.130@tcp41 [8/256/0/180] [15511.945118] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15511.976268] LNet: Added LNI 192.168.201.130@tcp42 [8/256/0/180] [15519.495765] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15519.527593] LNet: Added LNI 192.168.201.130@tcp43 [8/256/0/180] [15528.450761] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15528.474737] LNet: Added LNI 192.168.201.130@tcp44 [8/256/0/180] [15535.721903] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15535.755097] LNet: Added LNI 192.168.201.130@tcp45 [8/256/0/180] [15544.367828] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15544.411600] LNet: Added LNI 192.168.201.130@tcp46 [8/256/0/180] [15553.190362] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15553.228488] LNet: Added LNI 192.168.201.130@tcp47 [8/256/0/180] [15562.917836] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15562.989858] LNet: Added LNI 192.168.201.130@tcp48 [8/256/0/180] [15574.162544] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15574.190897] LNet: Added LNI 192.168.201.130@tcp49 [8/256/0/180] [15584.685531] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15584.764521] LNet: Added LNI 192.168.201.130@tcp50 [8/256/0/180] [15609.830218] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15609.836945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15609.857555] Lustre: Skipped 2 previous similar messages [15609.862818] Lustre: Skipped 1 previous similar message [15612.746428] Lustre: server umount lustre-MDT0000 complete [15616.627915] LustreError: 473549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786962587 with bad export cookie 16498305968834714294 [15616.632545] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15616.638694] LustreError: 473549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15617.238969] Lustre: server umount lustre-MDT0001 complete [15635.431184] Lustre: 472828:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786962590/real 1786962590] req@00000000c64f9134 x1873765630924800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786962606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15635.462438] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15636.173445] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15636.449674] Lustre: 472830:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786962591/real 1786962591] req@000000005363ef81 x1873765630924928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786962607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15640.543593] Lustre: 472831:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786962595/real 1786962595] req@00000000859a434e x1873765630925056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786962611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15642.591252] Lustre: 472829:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786962597/real 1786962597] req@000000003ed24acb x1873765630925312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786962613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15645.336964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15648.741176] Lustre: 472828:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786962603/real 1786962603] req@000000005b47199b x1873765630925440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786962619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15662.440813] LustreError: 475520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15662.456821] LustreError: 475520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15662.492557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15664.549390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15664.574594] LustreError: 475521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15666.594891] LustreError: 475520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15667.711453] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15671.790654] LustreError: 475521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15676.899309] LustreError: 475520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15676.955731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15677.281286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15680.374443] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15682.440807] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15691.088697] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15697.449957] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15716.325329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15716.353218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15716.364917] Lustre: Skipped 2 previous similar messages [15721.449636] LustreError: 475512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15721.481453] LustreError: 475512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15721.686527] Lustre: server umount lustre-OST0000 complete [15726.561785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15726.564889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15726.583432] Lustre: Skipped 2 previous similar messages [15726.594396] Lustre: Skipped 1 previous similar message [15731.682736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15731.689071] Lustre: Skipped 1 previous similar message [15734.683505] Lustre: server umount lustre-MDT0000 complete [15738.866469] LustreError: 474352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786962709 with bad export cookie 16498305968834716884 [15738.866984] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15738.873824] LustreError: 474352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15739.286401] Lustre: server umount lustre-MDT0001 complete [15750.677254] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15753.880992] Key type lgssc unregistered [15754.229917] LNet: 502559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15754.236296] LNetError: 502559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15754.242612] LNet: Removed LNI 192.168.201.130@tcp [15754.245615] LNet: Removed LNI 192.168.201.130@tcp10 [15754.248337] LNet: Removed LNI 192.168.201.130@tcp11 [15754.250355] LNet: Removed LNI 192.168.201.130@tcp12 [15754.254627] LNet: Removed LNI 192.168.201.130@tcp13 [15754.256379] LNet: Removed LNI 192.168.201.130@tcp14 [15754.257710] LNet: Removed LNI 192.168.201.130@tcp15 [15754.259776] LNet: Removed LNI 192.168.201.130@tcp16 [15754.261959] LNet: Removed LNI 192.168.201.130@tcp17 [15754.265487] LNet: Removed LNI 192.168.201.130@tcp18 [15754.268831] LNet: Removed LNI 192.168.201.130@tcp19 [15754.276417] LNet: Removed LNI 192.168.201.130@tcp20 [15754.283653] LNet: Removed LNI 192.168.201.130@tcp21 [15754.288727] LNet: Removed LNI 192.168.201.130@tcp22 [15754.292275] LNet: Removed LNI 192.168.201.130@tcp23 [15754.296516] LNet: Removed LNI 192.168.201.130@tcp24 [15754.298224] LNet: Removed LNI 192.168.201.130@tcp25 [15754.302677] LNet: Removed LNI 192.168.201.130@tcp26 [15754.305553] LNet: Removed LNI 192.168.201.130@tcp27 [15754.322333] LNet: Removed LNI 192.168.201.130@tcp28 [15754.328493] LNet: Removed LNI 192.168.201.130@tcp29 [15754.336714] LNet: Removed LNI 192.168.201.130@tcp30 [15754.343578] LNet: Removed LNI 192.168.201.130@tcp31 [15754.348457] LNet: Removed LNI 192.168.201.130@tcp32 [15754.352370] LNet: Removed LNI 192.168.201.130@tcp33 [15754.363425] LNet: Removed LNI 192.168.201.130@tcp34 [15754.366819] LNet: Removed LNI 192.168.201.130@tcp35 [15754.369975] LNet: Removed LNI 192.168.201.130@tcp36 [15754.372884] LNet: Removed LNI 192.168.201.130@tcp37 [15754.376629] LNet: Removed LNI 192.168.201.130@tcp38 [15754.378909] LNet: Removed LNI 192.168.201.130@tcp39 [15754.381384] LNet: Removed LNI 192.168.201.130@tcp40 [15754.384758] LNet: Removed LNI 192.168.201.130@tcp41 [15754.387310] LNet: Removed LNI 192.168.201.130@tcp42 [15754.389782] LNet: Removed LNI 192.168.201.130@tcp43 [15754.391987] LNet: Removed LNI 192.168.201.130@tcp44 [15754.395275] LNet: Removed LNI 192.168.201.130@tcp45 [15754.397751] LNet: Removed LNI 192.168.201.130@tcp46 [15754.400329] LNet: Removed LNI 192.168.201.130@tcp47 [15754.402738] LNet: Removed LNI 192.168.201.130@tcp48 [15754.405420] LNet: Removed LNI 192.168.201.130@tcp49 [15754.414726] LNet: Removed LNI 192.168.201.130@tcp50 [15755.235936] Key type .llcrypt unregistered [15755.244061] Key type ._llcrypt unregistered [15777.146700] Key type ._llcrypt registered [15777.151601] Key type .llcrypt registered [15777.291894] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [15793.040499] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15794.081450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15794.115348] alg: No test for adler32 (adler32-zlib) [15795.146549] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [15795.436367] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15797.103260] Key type lgssc registered [15798.592256] Lustre: Echo OBD driver; http://www.lustre.org/ [15834.706782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15834.724615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15836.111607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15836.158621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15836.212814] Lustre: lustre-MDT0000: new disk, initializing [15836.272894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15836.305569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15841.149419] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15851.396043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15851.541977] Lustre: 505757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15851.601769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15851.605390] Lustre: Skipped 1 previous similar message [15851.682425] Lustre: lustre-MDT0001: new disk, initializing [15851.768265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15851.817730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15851.827625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15856.643182] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15865.857637] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15872.044703] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15882.230878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15882.515275] Lustre: *** cfs_fail_loc=90f, val=0*** [15882.595310] Lustre: lustre-OST0000: new disk, initializing [15882.598960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15882.604327] Lustre: 507773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15882.663824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15889.382093] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15891.082704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15891.103297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15891.261782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15899.198465] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15911.906936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15911.909424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15911.918018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15917.029117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15917.046854] Lustre: Skipped 2 previous similar messages [15921.119235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15921.263351] Lustre: server umount lustre-OST0000 complete [15927.265056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15927.285736] Lustre: Skipped 1 previous similar message [15927.290877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15927.299077] Lustre: Skipped 1 previous similar message [15931.641438] Lustre: server umount lustre-MDT0000 complete [15935.962272] LustreError: 505749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786962906 with bad export cookie 15786607221126928694 [15935.979855] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15936.014683] LustreError: 503980:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [15936.341793] Lustre: server umount lustre-MDT0001 complete [15948.662497] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15951.664865] Key type lgssc unregistered [15952.060572] LNet: 509684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15952.067803] LNetError: 509684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15952.095483] LNet: Removed LNI 192.168.201.130@tcp [15953.178219] Key type .llcrypt unregistered [15953.180415] Key type ._llcrypt unregistered [15973.350301] Key type ._llcrypt registered [15973.352386] Key type .llcrypt registered [15973.450972] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15974.512157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15974.550860] alg: No test for adler32 (adler32-zlib) [15975.551607] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [15975.876234] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15977.647513] Key type lgssc registered [15978.867813] Lustre: Echo OBD driver; http://www.lustre.org/ [15985.428382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15985.463695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15987.000543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15991.246433] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15992.299194] LustreError: 510957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15992.315803] LustreError: 510957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15996.386029] LustreError: 510958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16000.086523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16000.322414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16004.394887] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16011.991927] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16016.985939] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16023.367520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16023.678852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16031.146657] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16038.031121] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16048.746283] loop0: detected capacity change from 0 to 204800000 [16048.985183] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16048.997753] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16049.008726] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16054.270892] loop0: detected capacity change from 0 to 204800000 [16054.473855] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16054.484371] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16054.493428] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16059.666986] loop0: detected capacity change from 0 to 204800000 [16065.741600] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16065.924804] Lustre: fs15246-OST0000: new disk, initializing [16065.928387] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16065.934544] Lustre: 514551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16065.989726] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16073.822932] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16084.228325] loop1: detected capacity change from 0 to 204800000 [16089.814079] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16089.923316] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16090.009816] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16090.155228] Lustre: fs15246-MDT0000: new disk, initializing [16090.395486] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16090.422504] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16090.533969] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16091.558482] Lustre: 515676:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000032343b8b x1873766458113152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786963078 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16094.940253] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16100.648634] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16100.661640] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16100.693932] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16117.732961] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16117.747252] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16122.505758] Lustre: server umount fs15246-MDT0000 complete [16122.876398] LustreError: 510953:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16127.980583] LustreError: 510952:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16133.093658] LustreError: 510954:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16133.688709] Lustre: server umount fs15246-OST0000 complete [16147.938664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16147.965539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16147.983585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16152.645926] Lustre: server umount lustre-OST0000 complete [16158.691191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16158.711216] Lustre: Skipped 1 previous similar message [16158.716594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16158.735026] Lustre: Skipped 3 previous similar messages [16162.587044] Lustre: server umount lustre-MDT0000 complete [16163.808370] LustreError: 513168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16163.838739] LustreError: 513168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16166.971959] LustreError: 510937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963137 with bad export cookie 6562756668811269286 [16166.973648] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16166.981692] LustreError: 510937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16167.666349] Lustre: server umount lustre-MDT0001 complete [16179.681300] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [16182.514989] Key type lgssc unregistered [16182.812923] LNet: 518775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16182.820116] LNetError: 518775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16182.840772] LNet: Removed LNI 192.168.201.130@tcp [16183.757664] Key type .llcrypt unregistered [16183.759348] Key type ._llcrypt unregistered [16201.638984] Key type ._llcrypt registered [16201.640963] Key type .llcrypt registered [16201.734921] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16203.108364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16203.156812] alg: No test for adler32 (adler32-zlib) [16204.231486] Lustre: Lustre: Build Version: 2.17.57_1_g72a0e6d [16204.461576] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [16206.135158] Key type lgssc registered [16207.454649] Lustre: Echo OBD driver; http://www.lustre.org/ [16247.148286] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [16257.888886] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16292.518358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16292.560915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16293.891978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16293.931464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16294.036783] Lustre: lustre-MDT0000: new disk, initializing [16294.101829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16294.123352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16298.333471] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16311.109204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16311.323838] Lustre: 523244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16311.425433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16311.432894] Lustre: Skipped 1 previous similar message [16311.605373] Lustre: lustre-MDT0001: new disk, initializing [16311.814568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16311.916202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16311.946409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16315.404809] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16326.607697] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16334.483492] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16342.394597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16342.712350] Lustre: lustre-OST0000: new disk, initializing [16342.716448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16342.725865] Lustre: 525212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16342.803192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16350.014725] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16351.350285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16351.374279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16351.539204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16360.717584] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16365.660994] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16365.812450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16370.608771] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16370.920858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16377.314660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16377.327198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16377.364082] LustreError: Skipped 1 previous similar message [16377.367328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16377.382807] Lustre: Skipped 1 previous similar message [16379.549804] Lustre: server umount lustre-OST0000 complete [16383.457507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16383.475851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16383.490458] Lustre: Skipped 1 previous similar message [16383.497374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16387.553847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16387.557834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16392.675055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16392.685660] Lustre: Skipped 2 previous similar messages [16397.279291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16397.514236] Lustre: server umount lustre-MDT0000 complete [16401.131411] LustreError: 523237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963371 with bad export cookie 12972758750030848118 [16401.140845] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16401.149433] LustreError: 523237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16401.544382] Lustre: server umount lustre-MDT0001 complete [16427.234963] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [16437.194077] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16467.990739] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16138 sec ========= 06:43:57 (1786963437) [16469.630333] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:43:59 (1786963439) === [16472.936896] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:44:02 (1786963442) === [16517.097051] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [16521.382900] Key type lgssc unregistered [16521.824820] LNet: 531862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16521.833224] LNetError: 531862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16521.855698] LNet: Removed LNI 192.168.201.130@tcp [16523.248742] Key type .llcrypt unregistered [16523.251347] Key type ._llcrypt unregistered