[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 530513142 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003386] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006017] kvm-guest: setup PV IPIs [ 0.009611] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012010] pid_max: default: 32768 minimum: 301 [ 0.013133] LSM: Security Framework initializing [ 0.014052] Yama: becoming mindful. [ 0.015040] SELinux: Initializing. [ 0.016085] *** VALIDATE selinux *** [ 0.023793] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027817] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029096] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031113] *** VALIDATE tmpfs *** [ 0.032472] *** VALIDATE proc *** [ 0.034234] *** VALIDATE cgroup *** [ 0.035019] *** VALIDATE cgroup2 *** [ 0.036289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043060] debug: unmapping init [mem 0xffffffff9ee59000-0xffffffff9ee60fff] [ 0.046167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054017] ... event mask: 000000070000000f [ 0.055284] rcu: Hierarchical SRCU implementation. [ 0.057504] smp: Bringing up secondary CPUs ... [ 0.058601] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062228] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137567] node 0 deferred pages initialised in 70ms [ 0.140107] devtmpfs: initialized [ 0.142115] x86/mm: Memory block size: 128MB [ 0.144583] gcov: version magic: 0x41383552 [ 0.147358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154373] pinctrl core: initialized pinctrl subsystem [ 0.156167] [ 0.156797] ************************************************************* [ 0.160016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162011] ** ** [ 0.165015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168013] ** ** [ 0.170016] ** This means that this kernel is built to expose internal ** [ 0.173015] ** IOMMU data structures, which may compromise security on ** [ 0.175018] ** your system. ** [ 0.178014] ** ** [ 0.180015] ** If you see this message and you are not debugging the ** [ 0.183015] ** kernel, report this immediately to your vendor! ** [ 0.185016] ** ** [ 0.188015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190015] ************************************************************* [ 0.193670] NET: Registered protocol family 16 [ 0.195371] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205489] cpuidle: using governor menu [ 0.206743] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207369] PCI: Using configuration type 1 for base access [ 0.209135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225033] cryptd: max_cpu_qlen set to 1000 [ 0.227292] ACPI: Added _OSI(Module Device) [ 0.229013] ACPI: Added _OSI(Processor Device) [ 0.230009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236053] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241433] ACPI: Interpreter enabled [ 0.242039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243008] ACPI: Using IOAPIC for interrupt routing [ 0.244076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256682] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268091] acpiphp: Slot [2] registered [ 0.269069] acpiphp: Slot [5] registered [ 0.270105] acpiphp: Slot [6] registered [ 0.272120] acpiphp: Slot [7] registered [ 0.273185] acpiphp: Slot [8] registered [ 0.275126] acpiphp: Slot [9] registered [ 0.276067] acpiphp: Slot [10] registered [ 0.277056] acpiphp: Slot [3] registered [ 0.277960] acpiphp: Slot [4] registered [ 0.279059] acpiphp: Slot [11] registered [ 0.280093] acpiphp: Slot [12] registered [ 0.281173] acpiphp: Slot [13] registered [ 0.282078] acpiphp: Slot [14] registered [ 0.283068] acpiphp: Slot [15] registered [ 0.284062] acpiphp: Slot [16] registered [ 0.286101] acpiphp: Slot [17] registered [ 0.287116] acpiphp: Slot [18] registered [ 0.288127] acpiphp: Slot [19] registered [ 0.290123] acpiphp: Slot [20] registered [ 0.291120] acpiphp: Slot [21] registered [ 0.293119] acpiphp: Slot [22] registered [ 0.294111] acpiphp: Slot [23] registered [ 0.296126] acpiphp: Slot [24] registered [ 0.298119] acpiphp: Slot [25] registered [ 0.299162] acpiphp: Slot [26] registered [ 0.301123] acpiphp: Slot [27] registered [ 0.302078] acpiphp: Slot [28] registered [ 0.303091] acpiphp: Slot [29] registered [ 0.304075] acpiphp: Slot [30] registered [ 0.305081] acpiphp: Slot [31] registered [ 0.306047] PCI host bridge to bus 0000:00 [ 0.307015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311067] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.318037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326996] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335841] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350465] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353836] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359248] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378939] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382089] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.412032] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.426027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.451068] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.465019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.488201] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.493027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.499021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.511023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.521207] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.527022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.538020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.566023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.587416] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.592022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.599032] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.625509] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627246] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628282] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.634520] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.643038] iommu: Default domain type: Passthrough [ 0.644347] SCSI subsystem initialized [ 0.646119] ACPI: bus type USB registered [ 0.647073] usbcore: registered new interface driver usbfs [ 0.649093] usbcore: registered new interface driver hub [ 0.650058] usbcore: registered new device driver usb [ 0.651133] pps_core: LinuxPPS API ver. 1 registered [ 0.652009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.654056] PTP clock support registered [ 0.656130] EDAC MC: Ver: 3.0.0 [ 0.658031] PCI: Using ACPI for IRQ routing [ 0.659843] NetLabel: Initializing [ 0.661016] NetLabel: domain hash size = 128 [ 0.663015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.666092] NetLabel: unlabeled traffic allowed by default [ 0.669042] vgaarb: loaded [ 0.670329] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.671011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.677000] clocksource: Switched to clocksource kvm-clock [ 0.784806] VFS: Disk quotas dquot_6.6.0 [ 0.786634] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.788573] *** VALIDATE ramfs *** [ 0.789527] *** VALIDATE hugetlbfs *** [ 0.790550] pnp: PnP ACPI init [ 0.793158] pnp: PnP ACPI: found 6 devices [ 0.807150] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810286] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812500] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.813903] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.816507] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.819139] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.822220] NET: Registered protocol family 2 [ 0.824558] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.829031] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.832526] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.837459] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.840781] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843975] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.847229] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850576] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853726] NET: Registered protocol family 1 [ 0.856493] RPC: Registered named UNIX socket transport module. [ 0.858906] RPC: Registered udp transport module. [ 0.860717] RPC: Registered tcp transport module. [ 0.862564] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865023] NET: Registered protocol family 44 [ 0.866712] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868298] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.870467] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.873065] PCI: CLS 0 bytes, default 64 [ 0.874818] Unpacking initramfs... [ 2.266181] debug: unmapping init [mem 0xffff8eebfcc54000-0xffff8eebfffbffff] [ 2.270569] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.273355] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.276188] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785959] Initialise system trusted keyrings [ 2.787879] Key type blacklist registered [ 2.789850] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.799264] zbud: loaded [ 2.802302] *** VALIDATE nfs *** [ 2.803614] *** VALIDATE nfs4 *** [ 2.805253] pstore: using deflate compression [ 2.809040] Platform Keyring initialized [ 2.902510] NET: Registered protocol family 38 [ 2.904502] Key type asymmetric registered [ 2.906164] Asymmetric key parser 'x509' registered [ 2.908198] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.911478] io scheduler mq-deadline registered [ 2.913429] io scheduler kyber registered [ 2.915284] io scheduler bfq registered [ 2.917818] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.921035] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.924044] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.927095] ACPI: Power Button [PWRF] [ 2.935594] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.943494] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.955316] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.960965] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.982167] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.006256] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.033710] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.037614] Non-volatile memory driver v1.3 [ 3.039030] Linux agpgart interface v0.103 [ 3.063833] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.065828] vda: detected capacity change from 0 to 74838016 [ 3.076771] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.078668] vdb: detected capacity change from 0 to 1073741824 [ 3.089428] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.091450] vdc: detected capacity change from 0 to 2621440000 [ 3.104965] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.107838] vdd: detected capacity change from 0 to 2621440000 [ 3.118786] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.121369] vde: detected capacity change from 0 to 4294967296 [ 3.134957] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.138054] vdf: detected capacity change from 0 to 4294967296 [ 3.144658] libphy: Fixed MDIO Bus: probed [ 3.155449] usbcore: registered new interface driver usbserial_generic [ 3.158261] usbserial: USB Serial support registered for generic [ 3.160927] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.165710] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.167726] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.170401] mousedev: PS/2 mouse device common for all mice [ 3.173818] rtc_cmos 00:05: RTC can wake from S4 [ 3.176409] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.181151] rtc_cmos 00:05: registered as rtc0 [ 3.183073] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.183926] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.190153] intel_pstate: CPU model not supported [ 3.190240] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.195818] hid: raw HID events driver (C) Jiri Kosina [ 3.198145] usbcore: registered new interface driver usbhid [ 3.199724] usbhid: USB HID core driver [ 3.200973] drop_monitor: Initializing network drop monitor service [ 3.202526] Initializing XFRM netlink socket [ 3.203882] NET: Registered protocol family 10 [ 3.206263] Segment Routing with IPv6 [ 3.207904] NET: Registered protocol family 17 [ 3.209363] mpls_gso: MPLS GSO support [ 3.214303] RAS: Correctable Errors collector initialized. [ 3.215982] AVX version of gcm_enc/dec engaged. [ 3.217217] AES CTR mode by8 optimization enabled [ 3.295913] sched_clock: Marking stable (3295881380, 0)->(4196117107, -900235727) [ 3.300531] registered taskstats version 1 [ 3.302805] Loading compiled-in X.509 certificates [ 3.305274] zswap: loaded using pool lzo/zbud [ 3.328271] Key type big_key registered [ 3.341281] Key type encrypted registered [ 3.343410] ima: No TPM chip found, activating TPM-bypass! [ 3.345934] ima: Allocated hash algorithm: sha1 [ 3.347930] ima: No architecture policies found [ 3.349978] evm: Initialising EVM extended attributes: [ 3.351729] evm: security.selinux [ 3.353149] evm: security.ima [ 3.354365] evm: security.capability [ 3.355846] evm: HMAC attrs: 0x1 [ 3.358578] rtc_cmos 00:05: setting system clock to 2026-08-24 07:19:43 UTC (1787555983) [ 3.366812] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.370844] debug: unmapping init [mem 0xffffffff9eb82000-0xffffffff9ee58fff] [ 3.378333] Write protecting the kernel read-only data: 28672k [ 3.383364] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.386735] debug: unmapping init [mem 0xffffffff9db14000-0xffffffff9dbfffff] [ 3.427205] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.436217] systemd[1]: Detected virtualization kvm. [ 3.438250] systemd[1]: Detected architecture x86-64. [ 3.440237] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.465705] systemd[1]: No hostname configured. [ 3.467602] systemd[1]: Set hostname to . [ 3.469779] random: systemd: uninitialized urandom read (16 bytes read) [ 3.472418] systemd[1]: Initializing machine ID from random generator. [ 3.508601] random: ln: uninitialized urandom read (6 bytes read) [ 3.627489] random: systemd: uninitialized urandom read (16 bytes read) [ 3.630736] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.637669] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.642369] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.241286] device-mapper: uevent: version 1.0.3 [ 4.243984] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 4.916375] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.996272] virtio_net virtio0 ens2: renamed from eth0 [ 5.072430] scsi host0: ata_piix [ 5.097912] scsi host1: ata_piix [ 5.104730] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.107165] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.560640] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.861809] random: crng init done [ 9.863069] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.391865] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.696656] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.954426] SELinux: Disabled at runtime. [ 12.021388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.030631] systemd[1]: Detected virtualization kvm. [ 12.032610] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.563264] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.566617] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.571616] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.574812] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.577419] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.587904] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.595666] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [FAILED] Failed to [ 12.726536] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 13.074166] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 13.463833] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.498628] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.656325] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.668332] EDAC sbridge: Ver: 1.1.2 [ 15.454108] Key type dns_resolver registered [ 15.737610] NFS: Registering the id_resolver key type [ 15.739595] Key type id_resolver registered [ 15.740902] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg223-server login: [ 44.419988] hrtimer: interrupt took 3635964 ns [ 51.644895] libcfs: loading out-of-tree module taints kernel. [ 51.731847] Key type ._llcrypt registered [ 51.733943] Key type .llcrypt registered [ 51.864740] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 71.947624] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 73.861265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.877686] alg: No test for adler32 (adler32-zlib) [ 75.382305] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 76.544956] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 78.271229] Key type lgssc registered [ 80.387473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.522780] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.358116] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 158.662396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.695636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.155882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.209910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.360217] Lustre: lustre-MDT0000: new disk, initializing [ 160.501258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.523288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.649794] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.326724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.422581] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.489771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.500156] Lustre: Skipped 1 previous similar message [ 181.628222] Lustre: lustre-MDT0001: new disk, initializing [ 181.738486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.793711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.822739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.647776] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.801795] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.118647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.472873] Lustre: lustre-OST0000: new disk, initializing [ 204.482725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.486797] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.576125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.927490] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.616792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.639302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.753227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.455466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.667441] Lustre: lustre-OST0001: new disk, initializing [ 229.672404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.680804] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.803090] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.115909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.127164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.258396] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.856394] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.721374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.203665] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.253435] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.432787] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing yml_node [ 277.393158] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 281.042966] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 284.213815] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 286.390294] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 03:24:24 EDT 2026 [ 310.152223] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 311.706283] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.207885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.211546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.246652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.769408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.772348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.777201] Lustre: Skipped 1 previous similar message [ 332.793874] Lustre: Skipped 3 previous similar messages [ 335.417504] Lustre: server umount lustre-MDT0000 complete [ 339.586458] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556319 with bad export cookie 12713043740163095454 [ 339.597477] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.609773] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.009484] LustreError: 12158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.016631] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.028230] LustreError: 12158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 343.029648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.041774] Lustre: Skipped 2 previous similar messages [ 343.075764] Lustre: Skipped 1 previous similar message [ 344.032446] LustreError: 9043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.064222] LustreError: 9043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 346.043937] Lustre: server umount lustre-MDT0001 complete [ 351.643667] Lustre: server umount lustre-OST0000 complete [ 356.089602] Lustre: server umount lustre-OST0001 complete [ 364.271982] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 375.407323] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 413.047612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.447325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.489967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.639807] Lustre: lustre-MDT0000: new disk, initializing [ 413.775239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.792853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.185306] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 431.767266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.945826] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.016206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.022345] Lustre: Skipped 1 previous similar message [ 432.117911] Lustre: lustre-MDT0001: new disk, initializing [ 432.227851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.279584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.298841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.096808] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 448.395860] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.628976] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.270558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.689403] Lustre: lustre-OST0000: new disk, initializing [ 464.701481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.717715] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.907309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.511574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.531783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.657900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.260386] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 483.983662] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.356913] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.630123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.049617] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.348864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.756840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.761499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.788711] Lustre: Skipped 1 previous similar message [ 502.817326] Lustre: Skipped 3 previous similar messages [ 507.880104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.991307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 513.174401] Lustre: server umount lustre-OST0000 complete [ 518.112280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.125926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.127843] Lustre: Skipped 3 previous similar messages [ 524.613440] Lustre: server umount lustre-MDT0000 complete [ 528.355542] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.367834] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 529.043723] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556509 with bad export cookie 12713043740163096987 [ 529.055024] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 529.066723] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.445465] Lustre: server umount lustre-MDT0001 complete [ 534.946322] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:28:33 (1787556513) [ 537.671839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 539.992994] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:28:38 (1787556518) [ 548.962707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.743742] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.887120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.976537] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.171473] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 564.194418] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.220855] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 567.522330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.506033] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 584.594658] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 592.140941] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.847527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 606.570721] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 615.424818] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 629.218532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 629.227635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.240674] Lustre: Skipped 1 previous similar message [ 629.246385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 629.255736] Lustre: Skipped 2 previous similar messages [ 630.260872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 631.691700] Lustre: server umount lustre-OST0000 complete [ 639.969276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 652.255670] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 652.537796] Lustre: server umount lustre-MDT0000 complete [ 655.842831] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.862485] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 658.224339] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556638 with bad export cookie 12713043740163097869 [ 658.256596] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.257748] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 659.343398] Lustre: server umount lustre-MDT0001 complete [ 678.890254] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 682.575864] Key type lgssc unregistered [ 683.041212] LNet: 23049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.054175] LNetError: 23049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.106393] LNet: Removed LNI 192.168.202.123@tcp [ 684.495986] Key type .llcrypt unregistered [ 684.498418] Key type ._llcrypt unregistered [ 711.385312] Key type ._llcrypt registered [ 711.389361] Key type .llcrypt registered [ 711.513674] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 713.360199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 713.595454] alg: No test for adler32 (adler32-zlib) [ 714.836437] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 715.190637] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 716.975248] Key type lgssc registered [ 718.528968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.142873] loop: module loaded [ 729.233790] loop0: detected capacity change from 0 to 204800000 [ 729.593914] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.606771] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.048100] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.650901] loop0: detected capacity change from 0 to 204800000 [ 737.866830] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.895122] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.924120] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.687287] loop0: detected capacity change from 0 to 204800000 [ 751.036466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.057275] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.532418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.596837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.710669] Lustre: lustre-MDT0000: new disk, initializing [ 752.789840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.806453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 758.575842] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 769.943774] loop1: detected capacity change from 0 to 204800000 [ 776.891367] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.506490] Lustre: lustre-OST07c6: new disk, initializing [ 777.521141] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 777.534163] Lustre: Skipped 1 previous similar message [ 777.556726] Lustre: 26238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 777.726771] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 783.970954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 783.985899] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 784.022055] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 784.874908] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 790.850817] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 790.862686] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 790.868620] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 790.874893] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 790.881379] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 790.888331] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 809.836743] Lustre: 25226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 809.854727] Lustre: 25226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 809.868707] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.880362] Lustre: 25226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 809.888349] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 809.900662] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 815.419652] Lustre: 25228:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 815.435242] Lustre: 25228:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 815.449637] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 815.463883] Lustre: 25228:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.482403] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 815.495878] Lustre: 25228:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.513527] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 815.527607] Lustre: 25228:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.542482] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 815.559283] Lustre: 25228:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.575737] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 815.593091] Lustre: 25228:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.858678] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 818.866842] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 818.873919] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 818.885442] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.895792] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 818.909143] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.922751] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 818.941931] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.951361] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.961982] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.974636] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.979541] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.026356] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 824.035019] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 824.042746] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 824.046046] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.059495] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.068681] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.079493] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 824.086468] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.094282] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 824.101091] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.107930] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 824.114512] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 835.321487] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 835.340815] Lustre: 26870:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 835.346157] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 835.350356] Lustre: 26870:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 835.356148] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 835.361632] Lustre: 26870:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 835.366145] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 835.369943] Lustre: 26870:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 835.377546] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 835.387301] Lustre: 26870:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 835.404112] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 835.412603] Lustre: 26870:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 886.241487] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.247126] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.262111] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 890.305915] Lustre: server umount lustre-OST07c6 complete [ 899.437279] Lustre: server umount lustre-MDT0000 complete [ 915.749376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 918.733399] Key type lgssc unregistered [ 919.110169] LNet: 28804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.128967] LNetError: 28804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.150189] LNet: Removed LNI 192.168.202.123@tcp [ 920.185988] Key type .llcrypt unregistered [ 920.190118] Key type ._llcrypt unregistered [ 939.621200] Key type ._llcrypt registered [ 939.623794] Key type .llcrypt registered [ 939.810404] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 941.640238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.700075] alg: No test for adler32 (adler32-zlib) [ 942.986677] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 943.538119] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 945.255159] Key type lgssc registered [ 946.772219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.839743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.865661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.642877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.344127] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 967.154881] LustreError: 30078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.170951] LustreError: 30078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 972.258064] LustreError: 30079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.371477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.654426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 978.760088] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 986.789295] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.196730] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 999.971133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1000.761287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.843660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1008.728180] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1018.486975] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1038.817809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.828712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1038.830088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.840581] Lustre: Skipped 1 previous similar message [ 1043.940854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1043.960392] Lustre: Skipped 1 previous similar message [ 1044.902903] Lustre: server umount lustre-OST0000 complete [ 1050.599405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1050.616540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.627222] Lustre: Skipped 1 previous similar message [ 1050.632802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.179306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.180993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.278469] Lustre: server umount lustre-MDT0000 complete [ 1059.307376] LustreError: 30078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.336600] LustreError: 30078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1061.404543] LustreError: 30058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557041 with bad export cookie 12704939843160432261 [ 1061.414325] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.423393] LustreError: 30058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1061.765962] Lustre: server umount lustre-MDT0001 complete [ 1074.287242] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1077.287411] Key type lgssc unregistered [ 1077.827795] LNet: 33821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.837046] LNetError: 33821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.865097] LNet: Removed LNI 192.168.202.123@tcp [ 1079.066371] Key type .llcrypt unregistered [ 1079.068468] Key type ._llcrypt unregistered [ 1101.237705] Key type ._llcrypt registered [ 1101.241989] Key type .llcrypt registered [ 1101.425832] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1103.158202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1103.202248] alg: No test for adler32 (adler32-zlib) [ 1104.365121] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1104.691996] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1106.367354] Key type lgssc registered [ 1108.286625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.560990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.589068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.195515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.328780] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1128.418614] LustreError: 35093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.452724] LustreError: 35093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1133.545978] LustreError: 35094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.742172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.195859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.449655] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1148.079724] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1155.769886] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1163.491545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1163.687431] LustreError: 36913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.042230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.416536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1171.403304] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1182.133235] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1194.979239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1194.987974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.026101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1196.001538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1196.001642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.487868] Lustre: server umount lustre-OST0000 complete [ 1206.242204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.243798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.263691] Lustre: Skipped 1 previous similar message [ 1206.280939] Lustre: Skipped 2 previous similar messages [ 1211.092632] Lustre: server umount lustre-MDT0000 complete [ 1211.374279] LustreError: 35093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.398577] LustreError: 35093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1215.098719] LustreError: 35074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557195 with bad export cookie 462015636322076876 [ 1215.107067] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.117041] LustreError: 35074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1215.471665] Lustre: server umount lustre-MDT0001 complete [ 1228.571157] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1230.912344] Key type lgssc unregistered [ 1231.186454] LNet: 38630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.194074] LNetError: 38630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.207921] LNet: Removed LNI 192.168.202.123@tcp [ 1232.277239] Key type .llcrypt unregistered [ 1232.279662] Key type ._llcrypt unregistered [ 1258.075351] Key type ._llcrypt registered [ 1258.077682] Key type .llcrypt registered [ 1258.158295] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1259.151819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.320135] alg: No test for adler32 (adler32-zlib) [ 1260.319734] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1260.536368] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1262.207161] Key type lgssc registered [ 1263.383637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.365830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.383952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.812957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.575843] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1283.049659] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.092688] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1288.173687] LustreError: 40046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.981084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.665427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.539763] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1304.367485] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1311.887657] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1318.495826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1318.952840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.771264] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1333.379045] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1343.544933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1350.273458] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 1351.141698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1351.152313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.162348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.357984] Lustre: server umount lustre-MDT0000 complete [ 1353.703647] LustreError: 41093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.706362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.079539] LustreError: 40026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557336 with bad export cookie 16863926047971739372 [ 1356.091667] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.816297] LustreError: 40042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.817934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.844412] LustreError: 40042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1358.862425] Lustre: Skipped 1 previous similar message [ 1358.866823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1361.824689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1361.830956] Lustre: Skipped 1 previous similar message [ 1362.374956] Lustre: server umount lustre-MDT0001 complete [ 1384.075332] Lustre: server umount lustre-OST0000 complete [ 1401.685648] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1404.618447] Key type lgssc unregistered [ 1404.949113] LNet: 43933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.955512] LNetError: 43933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.971682] LNet: Removed LNI 192.168.202.123@tcp [ 1405.730567] Key type .llcrypt unregistered [ 1405.732690] Key type ._llcrypt unregistered [ 1429.853303] Key type ._llcrypt registered [ 1429.858272] Key type .llcrypt registered [ 1429.995175] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1431.082599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.202267] alg: No test for adler32 (adler32-zlib) [ 1432.595572] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1432.832529] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1434.479170] Key type lgssc registered [ 1435.975769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1445.864682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1445.912557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.582737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.049391] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1457.192870] LustreError: 45349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.239308] LustreError: 45349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1462.241409] LustreError: 45350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.377208] LustreError: 45349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.485859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.230830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.586237] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1485.841072] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1492.267838] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1499.306925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.862526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.311654] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1514.014576] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1517.309313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1525.382988] Lustre: lustre-OST0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 1527.776276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1527.784200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.793806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1527.801337] Lustre: Skipped 1 previous similar message [ 1527.819275] Lustre: Skipped 1 previous similar message [ 1530.460343] Lustre: lustre-OST0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 1531.314239] Lustre: server umount lustre-OST0000 complete [ 1550.816526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.823100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.846947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.376506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.343810] Lustre: server umount lustre-MDT0000 complete [ 1558.503512] LustreError: 45346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.530771] LustreError: 45346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1561.982848] LustreError: 45331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557542 with bad export cookie 10616133214857024200 [ 1561.999490] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.727459] Lustre: server umount lustre-MDT0001 complete [ 1585.288553] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1588.894770] Key type lgssc unregistered [ 1589.226848] LNet: 49065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.233178] LNetError: 49065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.246873] LNet: Removed LNI 192.168.202.123@tcp [ 1589.968281] Key type .llcrypt unregistered [ 1589.970119] Key type ._llcrypt unregistered [ 1617.060362] Key type ._llcrypt registered [ 1617.062529] Key type .llcrypt registered [ 1617.205177] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1618.761253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1618.859443] alg: No test for adler32 (adler32-zlib) [ 1620.009565] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1620.433643] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1622.223331] Key type lgssc registered [ 1623.740974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.015830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1633.029502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.405250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.676875] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1644.525110] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.545339] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1649.650580] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.190343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.898167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1658.406845] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1670.591544] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.748396] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1687.499585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1688.079274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.498642] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1704.568924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1706.200627] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1713.883800] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1718.926341] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1730.714838] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 3h48m29s [ 1732.444820] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1734.765799] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 1735.659684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.707415] Lustre: Skipped 2 previous similar messages [ 1735.709932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.809756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.820936] Lustre: Skipped 5 previous similar messages [ 1740.881672] LustreError: 50480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.912661] Lustre: server umount lustre-MDT0000 complete [ 1743.344381] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.368635] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1744.422668] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1749.480391] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.499788] LustreError: 50484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1751.379701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.483241] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.722639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.794634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1752.768698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1757.169375] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1760.867487] Lustre: DEBUG MARKER: Wait for df (36747) ... [ 1762.547281] Lustre: DEBUG MARKER: done [ 1767.396443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.409344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.466601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.904188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.498789] Lustre: server umount lustre-OST0000 complete [ 1778.656682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.664583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1778.672482] Lustre: Skipped 2 previous similar messages [ 1778.703504] Lustre: Skipped 4 previous similar messages [ 1784.930887] Lustre: server umount lustre-MDT0000 complete [ 1785.892368] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.905258] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1789.681638] LustreError: 54456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557769 with bad export cookie 10615241392055091897 [ 1789.698602] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.705771] LustreError: 54456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1790.201424] Lustre: server umount lustre-MDT0001 complete [ 1801.905500] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1806.503154] Key type lgssc unregistered [ 1807.002604] LNet: 55414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.014676] LNetError: 55414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.038628] LNet: Removed LNI 192.168.202.123@tcp [ 1808.272055] Key type .llcrypt unregistered [ 1808.274331] Key type ._llcrypt unregistered [ 1860.599889] Key type ._llcrypt registered [ 1860.603390] Key type .llcrypt registered [ 1860.767760] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1862.714528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.750848] alg: No test for adler32 (adler32-zlib) [ 1864.035491] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1864.376114] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1866.111710] Key type lgssc registered [ 1867.842437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.279873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.303654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.728921] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1878.758524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1879.025453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.394604] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1897.109063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.211876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1897.250911] Lustre: 57619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1897.580583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.203796] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1913.223745] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1920.162872] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.828480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1928.060160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1928.432302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.783715] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1935.483131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1942.971758] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.003636] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 3h52m09s [ 1952.999645] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1962.701219] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1967.465841] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1967.618412] LustreError: 57627:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1987.722425] Lustre: lustre-MDT0000: Client ff75038b-f01f-4e17-90e9-7713d03eb6d1 (at 192.168.202.23@tcp) reconnecting [ 2007.671180] LustreError: 57627:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2010.307826] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2013.619349] Lustre: DEBUG MARKER: done [ 2022.370319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.376241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.414091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2022.881811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.921539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.004097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.020662] Lustre: Skipped 1 previous similar message [ 2033.122560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.131079] Lustre: Skipped 1 previous similar message [ 2035.167320] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2035.416266] Lustre: server umount lustre-OST0000 complete [ 2040.804655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2040.816755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.857727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.552870] Lustre: server umount lustre-MDT0000 complete [ 2049.610964] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558029 with bad export cookie 535795885688224583 [ 2049.614975] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.623716] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2050.364923] Lustre: server umount lustre-MDT0001 complete [ 2064.498817] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2068.601104] Key type lgssc unregistered [ 2069.198956] LNet: 61962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.206235] LNetError: 61962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.240937] LNet: Removed LNI 192.168.202.123@tcp [ 2070.665683] Key type .llcrypt unregistered [ 2070.668141] Key type ._llcrypt unregistered [ 2121.340226] Key type ._llcrypt registered [ 2121.343445] Key type .llcrypt registered [ 2121.442254] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2123.067255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.097624] alg: No test for adler32 (adler32-zlib) [ 2124.161845] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2124.347751] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2125.999409] Key type lgssc registered [ 2127.236618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2130.594566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2130.625029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.011738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2132.038974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2132.209581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.503975] Lustre: Failing over lustre-MDT0000 [ 2133.793055] Lustre: server umount lustre-MDT0000 complete [ 2141.367406] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:55:20 (1787558120) [ 2147.367783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.752068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.053796] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2163.372371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.574710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2163.642831] Lustre: 64765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2164.141875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.413469] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2179.024619] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.613907] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.350563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.597837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2194.795588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.861709] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2202.233714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2208.892121] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.643632] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2223.072684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.084114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.088438] Lustre: Skipped 1 previous similar message [ 2223.100391] Lustre: Skipped 2 previous similar messages [ 2227.836169] Lustre: server umount lustre-MDT0000 complete [ 2233.012137] LustreError: 64759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558213 with bad export cookie 1263752753747737768 [ 2233.016829] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.028187] LustreError: 64759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2233.316737] LustreError: 64775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.349443] LustreError: 64775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2233.368478] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.394433] Lustre: Skipped 1 previous similar message [ 2233.595898] Lustre: server umount lustre-MDT0001 complete [ 2237.701246] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2242.791289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2252.651680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.907076] LustreError: 63685:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000c056baa x1874390526502400/t0(0) o250->MGC192.168.202.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2258.624915] LustreError: 66616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.647600] LustreError: 66616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2258.718885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.849874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2263.811206] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2263.842398] LustreError: 66617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.845665] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2269.153375] LustreError: 66616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.076461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.325559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.351333] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2277.368956] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2285.262849] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2292.390674] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2299.883494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.902384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.993215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.008130] Lustre: Skipped 4 previous similar messages [ 2305.695258] Lustre: server umount lustre-MDT0000 complete [ 2309.495506] LustreError: 65503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558289 with bad export cookie 1263752753747740043 [ 2309.501457] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.508169] LustreError: 65503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2309.735831] Lustre: server umount lustre-MDT0001 complete [ 2317.044759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2325.689146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.496389] Lustre: 63687:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558290/real 1787558290] req@00000000c5d2c620 x1874390526529024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2326.525320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.535118] Lustre: Skipped 2 previous similar messages [ 2330.655238] Lustre: 63687:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558294/real 1787558294] req@0000000085efc461 x1874390526529152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2334.755676] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1189c06f130e4aee [ 2334.769042] Lustre: MGC192.168.202.123@tcp: Connection restored to 0@lo (at 0@lo) [ 2335.101608] LustreError: 69273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.178742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.262095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2336.163757] Lustre: 63689:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558299/real 1787558299] req@00000000f5ba4761 x1874390526529536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2336.197307] Lustre: 63689:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2339.681510] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2340.319472] Lustre: 63686:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558304/real 1787558304] req@00000000e934d8a1 x1874390526529664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2346.466334] LustreError: 71222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.485387] LustreError: 71222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2348.372142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.270413] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2354.163457] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2362.213611] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2370.319736] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.776860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.791369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.811987] Lustre: Skipped 2 previous similar messages [ 2380.848956] Lustre: Skipped 2 previous similar messages [ 2385.898092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.908440] Lustre: Skipped 2 previous similar messages [ 2386.681537] Lustre: server umount lustre-MDT0000 complete [ 2391.012485] LustreError: 71223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.065875] LustreError: 71223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2392.586409] LustreError: 64758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558372 with bad export cookie 1263752753747741422 [ 2392.606465] LustreError: 64758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2392.612650] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.489452] Lustre: server umount lustre-MDT0001 complete [ 2400.806662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2405.179829] Lustre: DEBUG MARKER: files compared the same [ 2417.885747] Lustre: server umount lustre-OST0000 complete [ 2439.566952] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2442.861197] Key type lgssc unregistered [ 2443.298803] LNet: 74501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.315944] LNetError: 74501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.352726] LNet: Removed LNI 192.168.202.123@tcp [ 2444.479807] Key type .llcrypt unregistered [ 2444.481507] Key type ._llcrypt unregistered [ 2469.656823] Key type ._llcrypt registered [ 2469.659835] Key type .llcrypt registered [ 2469.772725] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2470.793351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2470.965689] alg: No test for adler32 (adler32-zlib) [ 2471.977807] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2472.222633] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2473.871199] Key type lgssc registered [ 2474.979679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2481.998692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2482.023146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.394641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.650180] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug +malloc all [ 2492.898979] LustreError: 75930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.916642] LustreError: 75930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2496.248837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.588570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2501.246554] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug +malloc all [ 2509.569257] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2516.877639] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.134366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.552455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.681115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2530.384806] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug +malloc all [ 2537.879356] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2549.218428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.233127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.279026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.265879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2551.273181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.281061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.302804] Lustre: Skipped 1 previous similar message [ 2555.287540] Lustre: server umount lustre-OST0000 complete [ 2561.504487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.536319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2561.541622] Lustre: Skipped 1 previous similar message [ 2561.574615] Lustre: Skipped 1 previous similar message [ 2566.629967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.636624] Lustre: Skipped 1 previous similar message [ 2571.744874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.754180] Lustre: Skipped 1 previous similar message [ 2573.279213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2573.430466] Lustre: server umount lustre-MDT0000 complete [ 2576.871461] LustreError: 75926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.894746] LustreError: 75926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2577.222260] LustreError: 75911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558557 with bad export cookie 1229677193383302989 [ 2577.226706] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.235087] LustreError: 75911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2577.686796] Lustre: server umount lustre-MDT0001 complete [ 2589.896146] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2592.838344] Key type lgssc unregistered [ 2593.205413] LNet: 79470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.213095] LNetError: 79470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2593.232456] LNet: Removed LNI 192.168.202.123@tcp [ 2594.194189] Key type .llcrypt unregistered [ 2594.201761] Key type ._llcrypt unregistered [ 2625.643798] Key type ._llcrypt registered [ 2625.645400] Key type .llcrypt registered [ 2625.781084] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2627.460589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2627.528063] alg: No test for adler32 (adler32-zlib) [ 2628.717231] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2628.950881] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2630.671188] Key type lgssc registered [ 2631.513888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.605994] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2638.627564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.303571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2659.423349] LustreError: 80863:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.123@tcp: failed processing log, type 4: rc = -110 [ 2686.175285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.165114] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2702.921064] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2711.542995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.732870] Lustre: *** cfs_fail_loc=706, val=0*** [ 2712.157652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.235952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2716.833280] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2717.152713] LustreError: 80888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.171808] LustreError: 80888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2721.253560] LustreError: 81948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.270718] LustreError: 81948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2725.740263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.213325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.716296] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2742.888275] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.788423] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2758.112620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.126288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.155553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.218336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.225189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.238741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.281313] Lustre: Skipped 1 previous similar message [ 2763.985330] Lustre: server umount lustre-OST0000 complete [ 2772.455734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.463712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.485883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.492843] Lustre: Skipped 1 previous similar message [ 2775.233742] Lustre: server umount lustre-MDT0000 complete [ 2777.573981] LustreError: 81952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.600476] LustreError: 81952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2779.328674] LustreError: 80871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558759 with bad export cookie 13685603945878653434 [ 2779.338089] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.343329] LustreError: 80871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2779.816172] Lustre: server umount lustre-MDT0001 complete [ 2792.213697] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2794.939632] Key type lgssc unregistered [ 2795.205074] LNet: 84485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.218381] LNetError: 84485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.251855] LNet: Removed LNI 192.168.202.123@tcp [ 2796.519166] Key type .llcrypt unregistered [ 2796.522097] Key type ._llcrypt unregistered [ 2824.147360] Key type ._llcrypt registered [ 2824.148725] Key type .llcrypt registered [ 2824.304975] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2825.319995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2825.401916] alg: No test for adler32 (adler32-zlib) [ 2826.630594] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2826.838077] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2828.495197] Key type lgssc registered [ 2829.577861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.423564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2837.448915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.176923] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2850.992495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.483173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.922436] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2856.931826] LustreError: 86606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.957498] LustreError: 86606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2861.027809] LustreError: 86609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.049980] LustreError: 86609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2864.134815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.502617] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.521809] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2864.547087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.739372] LustreError: 86609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.772437] LustreError: 86609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2870.846358] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2874.850029] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.868681] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2880.367399] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.399546] Lustre: MGS: Client 54df002b-5c56-4195-b049-0c6f794f5983 (at 0@lo) reconnecting [ 2880.431179] LustreError: 86620:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2880.646090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.723440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2885.311937] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2901.472555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.483199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2901.491548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.932364] Lustre: server umount lustre-OST0000 complete [ 2911.716456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.719449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.731207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2911.734520] Lustre: Skipped 2 previous similar messages [ 2911.758016] Lustre: Skipped 2 previous similar messages [ 2916.148075] Lustre: server umount lustre-MDT0000 complete [ 2916.836723] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.885767] LustreError: 86610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2922.119938] Lustre: server umount lustre-MDT0001 complete [ 2925.815653] Lustre: server umount MGS complete [ 2939.451870] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:08:38 (1787558918) [ 2971.831487] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 2980.607566] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3021.501458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.371476] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3036.258245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.472645] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3036.516875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3036.588797] Lustre: lustre-MDT0001: new disk, initializing [ 3036.687503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.704404] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3036.709377] Lustre: 93794:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3036.719052] LustreError: 93794:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3040.982667] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3051.510624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.794581] Lustre: lustre-OST0000: new disk, initializing [ 3051.800845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3051.806908] Lustre: 94560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3051.872370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.185575] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3062.824833] Lustre: 95016:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3860026 x1874391262600960/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787559058 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3070.232749] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3077.970836] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.991096] Lustre: MGS: Client 47bb785c-aff1-4ca7-a738-9de94c59073f (at 0@lo) reconnecting [ 3078.053183] Lustre: 93060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3078.070793] LustreError: 93060:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3078.081605] LustreError: 93060:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3078.155425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3078.263524] Lustre: lustre-MDT0000: new disk, initializing [ 3078.514930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.535812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3078.558500] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3079.600046] Lustre: 95628:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fff48acb x1874391262607872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787559075 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3084.406657] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3084.778344] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3088.815595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3088.836769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3088.911893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3116.518719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.530222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.544924] Lustre: Skipped 1 previous similar message [ 3121.637487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.640017] Lustre: Skipped 1 previous similar message [ 3122.657767] Lustre: server umount lustre-OST0000 complete [ 3131.872903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3131.876930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.889948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3131.889956] Lustre: Skipped 1 previous similar message [ 3131.906340] Lustre: Skipped 1 previous similar message [ 3133.609223] Lustre: server umount lustre-MDT0000 complete [ 3136.997671] LustreError: 93765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.034271] LustreError: 93765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3143.622595] Lustre: server umount lustre-MDT0001 complete [ 3148.268984] Lustre: server umount MGS complete [ 3159.499741] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:12:18 (1787559138) [ 3177.731249] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 3180.601128] Key type lgssc unregistered [ 3180.923175] LNet: 98130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.935151] LNetError: 98130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.962230] LNet: Removed LNI 192.168.202.123@tcp [ 3181.768137] Key type .llcrypt unregistered [ 3181.771482] Key type ._llcrypt unregistered [ 3199.449328] Key type ._llcrypt registered [ 3199.451850] Key type .llcrypt registered [ 3199.638569] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3200.859112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3200.933750] alg: No test for adler32 (adler32-zlib) [ 3202.188878] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3202.456989] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 3204.151176] Key type lgssc registered [ 3205.398359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3213.883481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3213.911240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.521755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.249463] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3225.122263] LustreError: 99352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.138609] LustreError: 99352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3230.178830] LustreError: 99353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.832758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.312858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3237.390165] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3246.171712] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3252.123053] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3259.244595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3259.590027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.672594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3265.606173] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3276.815085] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3291.619628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3291.623134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.647376] LustreError: Skipped 1 previous similar message [ 3291.655639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.670793] Lustre: server umount lustre-OST0000 complete [ 3301.734688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.746434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3301.753815] Lustre: Skipped 2 previous similar messages [ 3301.771338] Lustre: Skipped 2 previous similar messages [ 3306.979997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3306.988087] Lustre: Skipped 1 previous similar message [ 3311.074140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.079287] Lustre: Skipped 1 previous similar message [ 3316.191417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3316.192276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.218922] Lustre: Skipped 1 previous similar message [ 3316.491405] Lustre: server umount lustre-MDT0000 complete [ 3320.489968] LustreError: 99333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559300 with bad export cookie 12279926533443575117 [ 3320.497317] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.500878] LustreError: 99333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3320.798747] Lustre: server umount lustre-MDT0001 complete [ 3332.120096] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 3335.203040] Key type lgssc unregistered [ 3335.463070] LNet: 102913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.469333] LNetError: 102913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3336.485804] LNet: Removed LNI 192.168.202.123@tcp [ 3337.349653] Key type .llcrypt unregistered [ 3337.351827] Key type ._llcrypt unregistered [ 3357.850556] Key type ._llcrypt registered [ 3357.852887] Key type .llcrypt registered [ 3357.962751] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3359.002903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3359.041454] alg: No test for adler32 (adler32-zlib) [ 3360.159761] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3360.500755] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 3362.191171] Key type lgssc registered [ 3363.477244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3369.601629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3369.609372] LustreError: 103999:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3369.612103] Lustre: Skipped 1 previous similar message [ 3369.612124] LustreError: 103996:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3369.625338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.626420] LustreError: 103999:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3369.626445] LustreError: 103999:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3369.648597] LustreError: 103999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3371.206806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.904057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.204166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3385.309292] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3392.410541] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing lsmod [ 3397.202689] LustreError: 105203:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3397.232974] LustreError: 105205:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3397.236985] LustreError: 105203:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3397.241948] LustreError: 105203:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3397.251998] LustreError: 105203:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3397.265728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.779815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.234159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.239371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.271565] Lustre: Skipped 2 previous similar messages [ 3403.303604] Lustre: Skipped 2 previous similar messages [ 3405.346542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.067961] Lustre: server umount lustre-MDT0000 complete [ 3410.403712] LustreError: 104032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.432333] LustreError: 104032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3413.259684] LustreError: 104007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559393 with bad export cookie 15280703323250548719 [ 3413.264464] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.281696] LustreError: 104007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3413.624658] LustreError: 104468:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001002f36c x1874391822522368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3414.231715] Lustre: server umount lustre-MDT0001 complete [ 3429.433756] Lustre: server umount lustre-OST0000 complete [ 3439.160415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.517327] LustreError: 106270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.621088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.150705] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3444.712466] LustreError: 106271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.839800] LustreError: 106270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.433385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.788813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.657337] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3468.561525] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3474.130132] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3482.631590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.366172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.423861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3491.095894] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3500.297927] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3509.216041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.224342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.238060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.244307] Lustre: Skipped 1 previous similar message [ 3509.250438] Lustre: Skipped 2 previous similar messages [ 3514.341689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.350359] Lustre: Skipped 2 previous similar messages [ 3514.754331] Lustre: server umount lustre-OST0000 complete [ 3519.463426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.477970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.489887] Lustre: Skipped 1 previous similar message [ 3519.519480] Lustre: Skipped 1 previous similar message [ 3524.787865] Lustre: server umount lustre-MDT0000 complete [ 3529.067180] LustreError: 106252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559509 with bad export cookie 15280703323250549419 [ 3529.068979] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.075700] LustreError: 106252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3529.387858] Lustre: server umount lustre-MDT0001 complete [ 3541.128621] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 3543.456475] Key type lgssc unregistered [ 3543.860507] LNet: 109804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3543.875091] LNetError: 109804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3543.910899] LNet: Removed LNI 192.168.202.123@tcp [ 3544.870898] Key type .llcrypt unregistered [ 3544.874563] Key type ._llcrypt unregistered [ 3571.505481] Key type ._llcrypt registered [ 3571.511770] Key type .llcrypt registered [ 3571.629825] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3572.824713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3572.849541] alg: No test for adler32 (adler32-zlib) [ 3573.870616] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3574.136448] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 3575.815351] Key type lgssc registered [ 3577.222129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3588.165282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3588.180764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.801311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.153459] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3599.331169] LustreError: 111225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.344568] LustreError: 111225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3604.448534] LustreError: 111224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.090771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.359744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.167987] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3621.526527] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3627.641801] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3636.728478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.288558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.032565] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3654.697526] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3659.858901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3664.583364] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3668.869529] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3675.104675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3675.117358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.138972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.176864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.178386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.185845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.205286] Lustre: Skipped 1 previous similar message [ 3683.302355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3687.391158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3687.595856] Lustre: server umount lustre-OST0000 complete [ 3693.537101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.555170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3693.559603] Lustre: Skipped 2 previous similar messages [ 3696.700761] Lustre: server umount lustre-MDT0000 complete [ 3698.656468] LustreError: 111220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.685369] LustreError: 111220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3700.977490] LustreError: 111204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559681 with bad export cookie 1160991632348710351 [ 3700.978517] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3700.987521] LustreError: 111204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3701.594876] Lustre: server umount lustre-MDT0001 complete [ 3713.011816] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 3715.714418] Key type lgssc unregistered [ 3716.194196] LNet: 114876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3716.201942] LNetError: 114876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3716.213808] LNet: Removed LNI 192.168.202.123@tcp [ 3717.271318] Key type .llcrypt unregistered [ 3717.273388] Key type ._llcrypt unregistered [ 3733.986544] Key type ._llcrypt registered [ 3733.988911] Key type .llcrypt registered [ 3734.073887] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3735.590312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3735.697205] alg: No test for adler32 (adler32-zlib) [ 3736.929809] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3737.247844] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 3738.967216] Key type lgssc registered [ 3740.949983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3750.658661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3750.689375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.097202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.796841] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3762.658455] LustreError: 116100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.681458] LustreError: 116100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3766.013568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.325243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3772.652793] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3781.983870] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3787.955903] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3794.598611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.957756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.080730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3801.159801] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3808.797903] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.049071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.056184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.056839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.064304] Lustre: Skipped 1 previous similar message [ 3825.128613] Lustre: server umount lustre-OST0000 complete [ 3832.288526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.293867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.299761] Lustre: Skipped 1 previous similar message [ 3832.336811] Lustre: Skipped 2 previous similar messages [ 3837.099866] Lustre: server umount lustre-MDT0000 complete [ 3837.417147] LustreError: 116097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.467258] LustreError: 116097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3842.542692] LustreError: 116096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.588188] LustreError: 116096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3842.656322] LustreError: 116081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559822 with bad export cookie 7646208168530834485 [ 3842.658822] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.673675] LustreError: 116081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3843.047682] Lustre: server umount lustre-MDT0001 complete [ 3857.216329] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 3859.987417] Key type lgssc unregistered [ 3860.320462] LNet: 119637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3860.332839] LNetError: 119637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3860.348808] LNet: Removed LNI 192.168.202.123@tcp [ 3861.404123] Key type .llcrypt unregistered [ 3861.406870] Key type ._llcrypt unregistered [ 3890.812843] Key type ._llcrypt registered [ 3890.817736] Key type .llcrypt registered [ 3890.939516] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3892.519318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3892.550210] alg: No test for adler32 (adler32-zlib) [ 3893.730930] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3893.907721] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 3895.567215] Key type lgssc registered [ 3896.931735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.094619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.118690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.630703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.731243] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3916.780079] LustreError: 121060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.802333] LustreError: 121060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3919.798394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.189161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.077416] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3931.703664] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3938.083841] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3945.786591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.254236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.374771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3954.250785] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3964.870476] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3972.863285] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3977.407050] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3977.416442] Lustre: Skipped 2 previous similar messages [ 3979.361890] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3979.405219] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3982.839912] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3982.844118] Lustre: Skipped 2 previous similar messages [ 3985.406409] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3985.413981] Lustre: Skipped 1 previous similar message [ 3992.902046] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3992.908357] Lustre: Skipped 2 previous similar messages [ 3994.209018] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.23@tcp 192.168.0.[2,10]@tcp [ 4000.339946] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.23/24@tcp in log lustre-MDT0000 [ 4000.358591] Lustre: Skipped 2 previous similar messages [ 4007.978732] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.23/24@tcp [ 4007.999640] Lustre: Skipped 1 previous similar message [ 4015.085393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4015.085515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.090299] LustreError: Skipped 2 previous similar messages [ 4015.121401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.120275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4017.123156] LustreError: Skipped 2 previous similar messages [ 4017.126940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.128237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.154654] Lustre: Skipped 1 previous similar message [ 4021.191254] Lustre: server umount lustre-OST0000 complete [ 4027.367967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.408963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.417559] Lustre: Skipped 1 previous similar message [ 4032.482590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.497322] Lustre: Skipped 1 previous similar message [ 4033.096031] Lustre: server umount lustre-MDT0000 complete [ 4037.213254] LustreError: 121041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560017 with bad export cookie 5490001451019514296 [ 4037.215451] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4037.240244] LustreError: 121041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4037.800501] Lustre: server umount lustre-MDT0001 complete [ 4052.662570] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 4056.571113] Key type lgssc unregistered [ 4057.024367] LNet: 125913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4057.032914] LNetError: 125913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4057.066992] LNet: Removed LNI 192.168.202.123@tcp [ 4058.232802] Key type .llcrypt unregistered [ 4058.251688] Key type ._llcrypt unregistered [ 4091.620735] Key type ._llcrypt registered [ 4091.624805] Key type .llcrypt registered [ 4091.745299] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4092.976516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4093.000620] alg: No test for adler32 (adler32-zlib) [ 4093.998812] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 4094.240201] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 4095.903318] Key type lgssc registered [ 4097.216373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4105.450701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4105.486474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.199142] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4107.225631] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.23/24@tcp [ 4107.257603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.490385] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4117.480653] LustreError: 127479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.524173] LustreError: 127479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4122.612829] LustreError: 127478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.070888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.393543] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4125.405769] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.23/24@tcp [ 4125.482977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.564147] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4142.537679] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4149.415485] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4156.242258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.611890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.016274] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4171.714745] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4177.198234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4190.690179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4190.699846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.728504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.741436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4192.748454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.751505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.792647] Lustre: Skipped 1 previous similar message [ 4195.408849] Lustre: server umount lustre-OST0000 complete [ 4202.464490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4202.484033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.524938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.898306] Lustre: server umount lustre-MDT0000 complete [ 4208.109152] LustreError: 128577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.714549] LustreError: 127461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560189 with bad export cookie 11528488901100591287 [ 4209.717478] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.721579] LustreError: 127461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4210.061899] Lustre: server umount lustre-MDT0001 complete [ 4220.057374] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 4222.474971] Key type lgssc unregistered [ 4222.790949] LNet: 131093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4222.799167] LNetError: 131093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4222.823766] LNet: Removed LNI 192.168.202.123@tcp [ 4223.506533] Key type .llcrypt unregistered [ 4223.508424] Key type ._llcrypt unregistered [ 4254.182953] Key type ._llcrypt registered [ 4254.184411] Key type .llcrypt registered [ 4254.338452] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 4275.527456] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4276.979426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.152387] alg: No test for adler32 (adler32-zlib) [ 4278.498258] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 4278.712341] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 4280.367985] Key type lgssc registered [ 4281.831994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4328.627512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4328.688303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.102854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4330.183584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4330.327847] Lustre: lustre-MDT0000: new disk, initializing [ 4330.430434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.451132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.382750] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4347.635077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.695362] Lustre: 134506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4347.736899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4347.740554] Lustre: Skipped 1 previous similar message [ 4347.840886] Lustre: lustre-MDT0001: new disk, initializing [ 4347.923311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4347.976920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4347.985522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4354.388740] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4367.651484] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4375.478034] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4385.682368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.109761] Lustre: lustre-OST0000: new disk, initializing [ 4386.114753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4386.138559] Lustre: 136475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4386.340149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.920561] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4394.062026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4394.084103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4394.246900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4404.157647] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.371650] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4409.538430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4414.960336] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4415.179599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4420.069793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4420.070684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.081028] LustreError: Skipped 1 previous similar message [ 4420.096140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4420.101877] Lustre: Skipped 1 previous similar message [ 4420.119315] Lustre: Skipped 1 previous similar message [ 4423.779618] Lustre: server umount lustre-OST0000 complete [ 4429.794889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4429.802661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.820203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4433.716951] Lustre: server umount lustre-MDT0000 complete [ 4437.181634] LustreError: 134500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560417 with bad export cookie 9305196869130122157 [ 4437.183308] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.192402] LustreError: 134500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4437.512470] Lustre: server umount lustre-MDT0001 complete [ 4446.396170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.872608] LustreError: 138691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.893438] LustreError: 138691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4446.934616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.211481] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4452.320704] LustreError: 138692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.442335] LustreError: 138691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.209127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.760552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4465.790104] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4473.305460] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4480.021348] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.900087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.486543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.531899] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4503.459537] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4508.337831] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4508.549338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4513.577678] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4513.724509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4522.606994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.852570] Lustre: lustre-OST0001: new disk, initializing [ 4522.857117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4522.867139] Lustre: 142063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4522.999608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4529.811489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4529.832173] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4530.094571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4531.298467] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4542.352710] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4542.768843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4548.689534] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4549.026458] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4555.602778] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9116cb32c000.ost_server_uuid 50 [ 4556.804616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9116cb32c000.ost_server_uuid in IDLE state after 0 sec [ 4571.103991] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4571.116373] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.127929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4571.140275] Lustre: Skipped 2 previous similar messages [ 4571.145679] Lustre: Skipped 3 previous similar messages [ 4576.269570] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4576.286308] Lustre: Skipped 1 previous similar message [ 4576.928848] Lustre: server umount lustre-OST0001 complete [ 4581.350779] LustreError: 140514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.374455] LustreError: 140514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4581.860044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.867340] LustreError: Skipped 1 previous similar message [ 4581.872787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.893403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4586.484817] LustreError: 141229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.511030] LustreError: 141229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4588.029912] Lustre: server umount lustre-OST0000 complete [ 4593.635406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.648321] LustreError: Skipped 1 previous similar message [ 4593.666087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.680500] Lustre: Skipped 1 previous similar message [ 4593.703402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.711580] Lustre: Skipped 3 previous similar messages [ 4599.380491] Lustre: server umount lustre-MDT0000 complete [ 4601.830780] LustreError: 140531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.881946] LustreError: 140531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4604.632524] LustreError: 138672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560584 with bad export cookie 9305196869130123046 [ 4604.640828] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.647806] LustreError: 138672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4605.136509] Lustre: server umount lustre-MDT0001 complete [ 4628.288245] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 4631.714382] Key type lgssc unregistered [ 4631.976694] LNet: 145095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4631.982716] LNetError: 145095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4631.991935] LNet: Removed LNI 192.168.202.123@tcp [ 4632.842777] Key type .llcrypt unregistered [ 4632.845296] Key type ._llcrypt unregistered [ 4678.938401] Key type ._llcrypt registered [ 4678.941939] Key type .llcrypt registered [ 4679.104991] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 4696.886803] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4698.396572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4698.440743] alg: No test for adler32 (adler32-zlib) [ 4699.544459] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 4699.815794] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 4701.471481] Key type lgssc registered [ 4702.805984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4740.736984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4740.770478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.033983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4742.067840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4742.164895] Lustre: lustre-MDT0000: new disk, initializing [ 4742.297654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4742.322480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4747.814315] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4761.011624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.123464] Lustre: 149253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4761.178612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4761.182212] Lustre: Skipped 1 previous similar message [ 4761.244816] Lustre: lustre-MDT0001: new disk, initializing [ 4761.378591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4761.435239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4761.448845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4768.175429] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4780.547322] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4788.317266] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4797.808525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.324754] Lustre: lustre-OST0000: new disk, initializing [ 4798.335379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4798.346225] Lustre: 151220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4798.462738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.147515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4804.153169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.339498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4805.523693] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4816.059595] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4821.728639] Lustre: Failing over lustre-OST0000 [ 4821.917709] Lustre: server umount lustre-OST0000 complete [ 4839.393543] Lustre: 147475:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560803/real 1787560803] req@000000007afd76c6 x1874393226579712/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787560819 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4839.418344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.441479] Lustre: 147477:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560805/real 1787560805] req@000000003b57910b x1874393226580608/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787560821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4841.505342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.845206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4842.213100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.236648] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4846.314300] Lustre: 147474:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560810/real 1787560810] req@00000000eb6e8ce5 x1874393226581888/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787560826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4846.337464] Lustre: 147474:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4849.388386] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4849.539985] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4851.423133] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560815/real 1787560815] req@00000000d296383b x1874393226583040/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787560831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4851.470402] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4851.566109] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4851.570543] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4853.593703] Lustre: Failing over lustre-MDT0000 [ 4853.737732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4853.766575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.800802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4854.065641] Lustre: server umount lustre-MDT0000 complete [ 4855.583126] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560819/real 1787560819] req@0000000097fa6392 x1874393226584064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787560835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4855.610033] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4859.873943] LustreError: 149258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.899897] LustreError: 149258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4861.014621] LustreError: 149259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.994591] LustreError: 149264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.010587] LustreError: 149264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4870.111163] Lustre: 147475:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560834/real 1787560834] req@000000005f32e804 x1874393226595456/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787560850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4870.118572] LustreError: 149258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.164764] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.196797] LustreError: 149258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4874.918283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.748714] LustreError: 149264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.781385] LustreError: 149264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4880.866089] LustreError: 147473:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005d283bf9 x1874393226600192/t0(0) o250->MGC192.168.202.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4881.245761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.328368] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4882.526492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4886.325783] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4886.511052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4886.522354] Lustre: Skipped 1 previous similar message [ 4886.539064] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4886.633364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4896.740084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.748608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4896.762458] Lustre: Skipped 3 previous similar messages [ 4900.747985] Lustre: server umount lustre-OST0000 complete [ 4906.978488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.982189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4906.990575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4906.990586] Lustre: Skipped 1 previous similar message [ 4912.112032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4912.124955] Lustre: Skipped 2 previous similar messages [ 4917.221561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4917.231899] Lustre: Skipped 1 previous similar message [ 4919.263170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4919.549360] Lustre: server umount lustre-MDT0000 complete [ 4922.341669] LustreError: 149259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.354406] LustreError: 149259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4923.314143] LustreError: 151219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560903 with bad export cookie 10575123442203227033 [ 4923.323919] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.327745] LustreError: 151219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4924.067782] Lustre: server umount lustre-MDT0001 complete [ 4937.727774] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 4941.159675] Key type lgssc unregistered [ 4941.419569] LNet: 154883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4941.422443] LNetError: 154883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4941.444121] LNet: Removed LNI 192.168.202.123@tcp [ 4942.611743] Key type .llcrypt unregistered [ 4942.613360] Key type ._llcrypt unregistered [ 4981.438996] Key type ._llcrypt registered [ 4981.441477] Key type .llcrypt registered [ 4981.547061] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4982.455026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4982.687519] alg: No test for adler32 (adler32-zlib) [ 4983.687340] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 4983.949093] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 4985.575238] Key type lgssc registered [ 4987.263656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4995.590135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4995.606521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.261418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.922539] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5007.852791] LustreError: 156406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.877677] LustreError: 156406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5012.180994] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5012.963702] LustreError: 156407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.508526] LustreError: 156435:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5017.532194] LustreError: 156435:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5017.557540] LustreError: 156435:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 5018.122178] Lustre: server umount lustre-MDT0000 complete [ 5037.070901] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 5049.391723] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 5088.128470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.530769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5088.581300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5088.656239] Lustre: lustre-MDT0000: new disk, initializing [ 5088.806489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.853125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5093.434249] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5104.827355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.987803] Lustre: 159741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5105.053820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5105.057717] Lustre: Skipped 1 previous similar message [ 5105.128581] Lustre: lustre-MDT0001: new disk, initializing [ 5105.230869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5105.280160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5105.301580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5110.059438] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5120.251781] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5126.854544] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5134.031088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.252784] Lustre: lustre-OST0000: new disk, initializing [ 5134.257116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5134.263595] Lustre: 161708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5134.366191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.416958] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5142.640340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5142.669655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5142.873467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5151.315582] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5156.929883] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5159.347389] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:45:37 (1787561137) [ 5427.689809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5427.700712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.721059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5429.728144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.729061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5429.736875] Lustre: Skipped 1 previous similar message [ 5429.752916] Lustre: Skipped 2 previous similar messages [ 5432.247625] Lustre: server umount lustre-MDT0000 complete [ 5434.856133] LustreError: 159747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.891125] LustreError: 159747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5436.675024] LustreError: 159732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561416 with bad export cookie 12425702284945635039 [ 5436.681079] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.690131] LustreError: 159732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5439.967735] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.971660] LustreError: 163498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.990931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5442.017629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5443.224387] Lustre: server umount lustre-MDT0001 complete [ 5455.490102] Lustre: server umount lustre-OST0000 complete [ 5467.914335] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 5480.665486] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 5525.954530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.416595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5526.464347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5526.645814] Lustre: lustre-MDT0000: new disk, initializing [ 5526.763200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.795630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5532.492533] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5545.063942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.212908] Lustre: 166387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5545.258524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5545.263380] Lustre: Skipped 1 previous similar message [ 5545.409723] Lustre: lustre-MDT0001: new disk, initializing [ 5545.568621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.619606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5545.635362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5550.989708] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5562.043665] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5569.505317] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5577.979443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5578.427667] Lustre: lustre-OST0000: new disk, initializing [ 5578.433750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5578.438550] Lustre: 168358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.520481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.997926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5584.011341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5584.182137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5585.238270] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5595.854286] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5602.147135] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5602.346657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5607.805108] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5608.116865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5615.080865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5615.082703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.090883] LustreError: Skipped 1 previous similar message [ 5615.103583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5615.109433] Lustre: Skipped 1 previous similar message [ 5615.130589] Lustre: Skipped 1 previous similar message [ 5616.600538] Lustre: server umount lustre-OST0000 complete [ 5620.192317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.313439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.318092] Lustre: Skipped 3 previous similar messages [ 5626.530928] Lustre: server umount lustre-MDT0000 complete [ 5629.960693] LustreError: 166379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561610 with bad export cookie 12425702284945793876 [ 5629.971133] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.977471] LustreError: 166379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.283151] Lustre: server umount lustre-MDT0001 complete [ 5639.759379] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:53:38 (1787561618) [ 5657.485388] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 5668.890560] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 5703.079898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.331233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5703.362330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5703.419263] Lustre: lustre-MDT0000: new disk, initializing [ 5703.481067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.495484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5708.595427] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5720.820446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.019298] Lustre: 173113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5721.437564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5726.701038] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5736.190247] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5742.965658] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5751.915816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5752.340336] Lustre: lustre-OST0000: new disk, initializing [ 5752.343250] Lustre: Skipped 1 previous similar message [ 5752.349850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5752.353898] Lustre: Skipped 2 previous similar messages [ 5752.360581] Lustre: 175093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5759.555358] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5762.140899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5762.152460] Lustre: Skipped 1 previous similar message [ 5762.161696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5762.248701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5769.591287] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5777.892073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.904974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5777.915048] Lustre: Skipped 1 previous similar message [ 5777.944651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.951301] Lustre: Skipped 1 previous similar message [ 5783.706440] Lustre: server umount lustre-OST0000 complete [ 5788.133302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.831453] Lustre: server umount lustre-MDT0000 complete [ 5797.761298] LustreError: 173915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561777 with bad export cookie 12425702284945794758 [ 5797.773952] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.781475] LustreError: 173915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5798.077108] Lustre: server umount lustre-MDT0001 complete [ 5806.418797] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:56:25 (1787561785) [ 5824.322606] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 5835.578726] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 5875.569633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.811396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5875.829109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5875.889643] Lustre: lustre-MDT0000: new disk, initializing [ 5875.936516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.938791] Lustre: Skipped 2 previous similar messages [ 5875.951290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5881.736725] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5893.739028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.051140] Lustre: 179415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5894.656878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5900.542363] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5912.031758] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5918.595780] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5926.700216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.111414] Lustre: 181395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5933.617580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5933.820705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5935.014234] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5947.956419] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5959.652481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5959.654573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.667166] LustreError: Skipped 1 previous similar message [ 5959.667797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.667801] Lustre: Skipped 6 previous similar messages [ 5959.714430] Lustre: Skipped 4 previous similar messages [ 5970.399175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5970.537603] Lustre: server umount lustre-OST0000 complete [ 5981.567750] Lustre: server umount lustre-MDT0000 complete [ 5985.254169] LustreError: 179428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.278380] LustreError: 179428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5987.284301] LustreError: 179407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561967 with bad export cookie 12425702284945795836 [ 5987.302848] LustreError: 179407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5987.310323] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.939794] Lustre: server umount lustre-MDT0001 complete [ 6006.716543] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 6010.847868] Key type lgssc unregistered [ 6011.284468] LNet: 183277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6011.302064] LNetError: 183277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6011.328344] LNet: Removed LNI 192.168.202.123@tcp [ 6012.641996] Key type .llcrypt unregistered [ 6012.645296] Key type ._llcrypt unregistered [ 6040.664074] Key type ._llcrypt registered [ 6040.667726] Key type .llcrypt registered [ 6040.833791] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 6042.895866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6043.059479] alg: No test for adler32 (adler32-zlib) [ 6044.264567] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6044.566369] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 6046.287833] Key type lgssc registered [ 6048.399679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.720529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6057.798211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.465334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.288639] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6069.744527] LustreError: 184696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.779518] LustreError: 184696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6074.848348] LustreError: 184695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.817640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.866377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6084.607339] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6093.646380] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.986924] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6107.598513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.110680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6116.801272] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6125.926636] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6129.715507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6148.064443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.068707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.075049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6150.115124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6150.120818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6150.123627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.159673] Lustre: Skipped 1 previous similar message [ 6154.319701] Lustre: server umount lustre-OST0000 complete [ 6159.840219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6159.850719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.875513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6164.328890] Lustre: server umount lustre-MDT0000 complete [ 6165.472625] LustreError: 184695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.494049] LustreError: 184695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6167.786115] LustreError: 184677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562147 with bad export cookie 4521942956538294302 [ 6167.805342] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.128484] Lustre: server umount lustre-MDT0001 complete [ 6179.993058] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 6182.761629] Key type lgssc unregistered [ 6183.042459] LNet: 188267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6183.055971] LNetError: 188267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.103454] LNet: Removed LNI 192.168.202.123@tcp [ 6185.118132] Key type .llcrypt unregistered [ 6185.122777] Key type ._llcrypt unregistered [ 6211.821419] Key type ._llcrypt registered [ 6211.825877] Key type .llcrypt registered [ 6211.936230] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 6213.147018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6213.239062] alg: No test for adler32 (adler32-zlib) [ 6214.341941] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6214.608747] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 6216.343171] Key type lgssc registered [ 6217.638865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6226.545165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6226.576281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.426276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6234.591531] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6238.696485] LustreError: 189685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6238.711344] LustreError: 189685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6243.706392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.812259] LustreError: 189686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.120614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6249.004525] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6257.468265] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6264.200127] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6271.916976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.334381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.341662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6279.027063] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6288.260449] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.104867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6299.107616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.126479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.809608] Lustre: lustre-OST0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 6301.155768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6301.179842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.702590] Lustre: server umount lustre-OST0000 complete [ 6314.342770] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9116d726f000.ost_server_uuid 50 [ 6317.912455] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6336.622177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6336.630028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.666610] Lustre: server umount lustre-MDT0000 complete [ 6336.667924] LustreError: 189686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.098486] LustreError: 189680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.119538] LustreError: 189680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6341.167659] LustreError: 189667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562321 with bad export cookie 6302213698841623420 [ 6341.171667] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6341.180694] LustreError: 189667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6341.772224] Lustre: server umount lustre-MDT0001 complete [ 6352.029878] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:05:30 (1787562330) [ 6360.302133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.974989] LustreError: 193599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.050893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.853533] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6371.301988] LustreError: 193600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.324421] LustreError: 193600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6375.588981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.116697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6380.598989] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6388.940842] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6396.020237] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6403.490475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.781815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.875371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6411.417488] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6423.178944] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.253200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.520744] Lustre: lustre-OST0001: new disk, initializing [ 6431.528313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6431.544533] Lustre: 196442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6431.676624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.584205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6438.604014] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6438.753069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6440.401885] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6451.431125] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6457.317713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.320258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.347732] Lustre: Skipped 1 previous similar message [ 6457.350842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6457.350850] Lustre: Skipped 2 previous similar messages [ 6460.910177] Lustre: server umount lustre-OST0000 complete [ 6462.437861] LustreError: 195421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.453705] LustreError: 195421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6467.146620] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6467.454830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6473.376612] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6473.602419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6490.080090] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.080372] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.092657] LustreError: Skipped 1 previous similar message [ 6490.098978] Lustre: Skipped 1 previous similar message [ 6490.101866] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6490.112783] Lustre: Skipped 3 previous similar messages [ 6495.534547] Lustre: server umount lustre-OST0001 complete [ 6503.394767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.417882] Lustre: Skipped 1 previous similar message [ 6503.434472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6503.437383] Lustre: Skipped 3 previous similar messages [ 6506.689498] Lustre: server umount lustre-MDT0000 complete [ 6508.516620] LustreError: 193600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.541296] LustreError: 193600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 6510.855047] LustreError: 193580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562490 with bad export cookie 6302213698841624491 [ 6510.855613] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6510.880977] LustreError: 193580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6511.253719] Lustre: server umount lustre-MDT0001 complete [ 6545.013696] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:08:42 (1787562522) [ 6553.831135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.613582] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6554.699723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6555.132285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.083649] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6570.938974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.109669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6571.200372] Lustre: 199891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6576.005053] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6584.835809] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6591.252541] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6600.035667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.304935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6608.148356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6609.356382] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6618.932878] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6627.622117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.715475] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6627.994100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6627.998090] Lustre: Skipped 2 previous similar messages [ 6635.070927] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6643.443594] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6648.812554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.818201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6648.822895] Lustre: Skipped 2 previous similar messages [ 6648.831403] Lustre: Skipped 1 previous similar message [ 6652.465721] Lustre: server umount lustre-OST0000 complete [ 6653.926544] LustreError: 201741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.940267] LustreError: 201741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6669.287155] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.296804] LustreError: Skipped 1 previous similar message [ 6669.297388] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6669.328751] Lustre: Skipped 2 previous similar messages [ 6674.442065] Lustre: server umount lustre-OST0001 complete [ 6684.136333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.161220] Lustre: Skipped 2 previous similar messages [ 6694.367896] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6694.791566] Lustre: server umount lustre-MDT0000 complete [ 6699.965469] LustreError: 199881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562680 with bad export cookie 6302213698841625898 [ 6699.970729] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6700.001203] LustreError: 199881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6700.884603] Lustre: server umount lustre-MDT0001 complete [ 6730.811886] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:11:49 (1787562709) [ 6750.784142] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 6761.277423] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 6812.869930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.229311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6813.279806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6813.369308] Lustre: lustre-MDT0000: new disk, initializing [ 6813.454861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.471475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6819.308842] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6832.344836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.476822] Lustre: 207993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6832.609456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6832.612244] Lustre: Skipped 1 previous similar message [ 6832.706820] Lustre: lustre-MDT0001: new disk, initializing [ 6832.909401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6832.929923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6839.966296] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6852.494570] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.121853] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.497750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.814514] Lustre: lustre-OST0000: new disk, initializing [ 6867.819017] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6867.829714] Lustre: 209966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6873.726860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6873.743924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6873.939289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6874.901870] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6887.686579] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6895.505474] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6895.934789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6902.122554] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6902.454800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6904.801966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6904.807328] LustreError: Skipped 1 previous similar message [ 6904.809799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.811313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6904.820318] Lustre: Skipped 2 previous similar messages [ 6904.829207] Lustre: Skipped 8 previous similar messages [ 6910.466491] Lustre: server umount lustre-OST0000 complete [ 6929.376602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6929.810421] Lustre: server umount lustre-MDT0000 complete [ 6930.413546] LustreError: 208000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6930.427777] LustreError: 208000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6933.683912] LustreError: 207985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562913 with bad export cookie 6302213698841627459 [ 6933.700390] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.711632] LustreError: 207985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6934.200720] Lustre: server umount lustre-MDT0001 complete [ 6943.234245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.415348] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6961.293544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.824658] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6978.653551] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.355223] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6994.154753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.217624] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7011.293382] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7017.680434] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7018.093192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7023.536771] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7023.776039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7027.680179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7027.686853] LustreError: Skipped 1 previous similar message [ 7027.692378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7027.702340] Lustre: Skipped 6 previous similar messages [ 7032.429229] Lustre: server umount lustre-OST0000 complete [ 7041.649110] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7048.109584] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7080.741356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.493489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.496589] Lustre: Skipped 5 previous similar messages [ 7092.095359] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7093.737252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7093.764757] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7103.020791] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7103.051175] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7103.989355] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7106.626823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7113.211244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.217319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.245093] Lustre: Skipped 4 previous similar messages [ 7113.259771] LustreError: Skipped 1 previous similar message [ 7117.030955] Lustre: server umount lustre-OST0000 complete [ 7128.025567] Lustre: server umount lustre-MDT0000 complete [ 7132.563482] LustreError: 212166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563112 with bad export cookie 6302213698841628341 [ 7132.581753] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.588063] LustreError: 212166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7133.049169] Lustre: server umount lustre-MDT0001 complete [ 7146.187684] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:18:43 (1787563123) [ 7156.494498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.718665] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7174.194814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.651061] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7190.995224] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7200.387824] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7209.786137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.702870] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7230.301928] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.688879] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7235.926292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7241.817631] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7242.167601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7249.607213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.792584] Lustre: lustre-OST0001: new disk, initializing [ 7249.796807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7249.812656] Lustre: 221532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7257.208045] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7257.705471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7257.715687] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7257.771758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7267.762877] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.872522] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7274.093690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7279.825448] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7280.160283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7283.721879] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.734742] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7283.751154] LustreError: Skipped 1 previous similar message [ 7283.771358] Lustre: Skipped 6 previous similar messages [ 7288.815319] LustreError: 221910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.853030] LustreError: 221910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7289.088221] Lustre: server umount lustre-OST0001 complete [ 7297.387199] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7297.698746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7303.249654] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7303.547729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7335.895857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.548592] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7341.575091] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7341.576331] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7342.481485] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7352.046687] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7353.597469] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7370.207481] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7370.325830] Lustre: server umount lustre-OST0001 complete [ 7389.664242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7389.679227] Lustre: Skipped 7 previous similar messages [ 7394.357355] Lustre: server umount lustre-OST0000 complete [ 7404.937828] Lustre: server umount lustre-MDT0000 complete [ 7411.264891] LustreError: 218141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563391 with bad export cookie 6302213698841629300 [ 7411.273210] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7411.285024] LustreError: 218141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7417.962470] Lustre: server umount lustre-MDT0001 complete [ 7453.547037] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:23:52 (1787563432) [ 7461.782798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7462.322584] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7462.374171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7468.557192] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7479.088918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.194173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7479.228364] Lustre: 226782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7484.183037] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7493.045627] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7499.335522] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7507.543711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7507.877396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7515.053323] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7524.339363] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7536.087432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7536.202996] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7543.720225] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7553.421144] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7555.747472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7560.989574] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7561.254320] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7567.040840] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7567.485352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7575.003675] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9116d084a000.ost_server_uuid 50 [ 7577.379955] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9116d084a000.ost_server_uuid in IDLE state after 0 sec [ 7578.897879] Lustre: Permanently deactivating lustre-OST0001 [ 7578.911985] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7583.441780] Lustre: Permanently reactivating lustre-OST0001 [ 7583.461321] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7583.471275] Lustre: Skipped 2 previous similar messages [ 7590.880701] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7590.887402] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7590.893219] LustreError: Skipped 6 previous similar messages [ 7590.912488] Lustre: Skipped 15 previous similar messages [ 7595.437619] Lustre: server umount lustre-OST0001 complete [ 7605.743995] Lustre: server umount lustre-OST0000 complete [ 7616.612687] Lustre: server umount lustre-MDT0000 complete [ 7620.779635] LustreError: 228644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563600 with bad export cookie 6302213698841630595 [ 7620.785328] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7620.796140] LustreError: 228644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7621.509678] Lustre: server umount lustre-MDT0001 complete [ 7647.900612] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:27:06 (1787563626) [ 7657.682835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.053864] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7658.107300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7658.114832] Lustre: Skipped 2 previous similar messages [ 7658.290810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7658.304773] Lustre: Skipped 9 previous similar messages [ 7664.243466] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7674.709535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.842812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7674.883547] Lustre: 233482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7681.527960] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7691.737464] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7699.382140] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7708.161461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.418364] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7708.450059] Lustre: Permanently deactivating lustre-OST0000 [ 7708.467905] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7716.391819] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7716.619558] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7725.567316] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7733.434870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7733.492536] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7740.582406] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7740.659508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7749.286288] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7752.025723] Lustre: Permanently reactivating lustre-OST0000 [ 7752.096930] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7752.110182] Lustre: Skipped 2 previous similar messages [ 7760.986161] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 000000000b2ba8ed, cur 1787563741 deadline 1787563726 last 1787563696 [ 7761.667239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7770.444882] Lustre: server umount lustre-OST0001 complete [ 7780.734454] Lustre: server umount lustre-OST0000 complete [ 7799.263257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7799.467279] Lustre: server umount lustre-MDT0000 complete [ 7801.313064] LustreError: 234509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.322108] LustreError: 234509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 48 previous similar messages [ 7804.461642] LustreError: 233474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563784 with bad export cookie 6302213698841632268 [ 7804.466501] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.470868] LustreError: 233474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7804.850172] Lustre: server umount lustre-MDT0001 complete [ 7820.580074] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 7824.746267] Key type lgssc unregistered [ 7825.338700] LNet: 238347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7825.379697] LNetError: 238347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7826.497835] LNet: Removed LNI 192.168.202.123@tcp [ 7827.622680] Key type .llcrypt unregistered [ 7827.627312] Key type ._llcrypt unregistered [ 7856.170587] Key type ._llcrypt registered [ 7856.172584] Key type .llcrypt registered [ 7856.280843] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7857.561448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7857.666522] alg: No test for adler32 (adler32-zlib) [ 7858.704708] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 7858.952115] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 7860.623580] Key type lgssc registered [ 7862.489650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7872.366337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.398276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.916825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7879.104276] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7883.240143] LustreError: 239814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.261978] LustreError: 239814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7884.257632] LustreError: 239815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.241958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.366158] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7889.380087] LustreError: 239814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.428561] Lustre: 239804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7889.442499] Lustre: Permanently deactivating lustre-MDT0001 [ 7889.454374] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7889.744664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7894.726727] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7898.179904] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7904.056305] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7909.868771] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7917.917987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7918.303434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.338953] LustreError: 241656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.371952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7924.715309] LustreError: 241656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7925.373852] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7933.928908] LustreError: 241657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.970742] LustreError: 241657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7934.378510] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.597512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.968980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7947.290181] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7947.297377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7949.415699] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7957.183618] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7961.392791] Lustre: Permanently reactivating lustre-MDT0001 [ 7961.408740] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7961.412654] Lustre: Skipped 2 previous similar messages [ 7962.645506] Lustre: lustre-MDT0001: Client 76816806-8127-4038-87a9-f32b6ae1e70c (at 192.168.202.23@tcp) reconnecting [ 7968.947908] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.963528] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7968.978536] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7969.004311] LustreError: 243554:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7969.004606] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7969.023804] LustreError: 243554:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7969.023937] LustreError: 243554:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.123@tcp: cfg command failed: rc = -108 [ 7969.043962] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.053227] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7969.053227] [ 7969.053628] LustreError: 239806:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7970.896647] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7976.889198] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7977.084826] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7982.674407] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7982.915399] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7985.850333] Lustre: Permanently deactivating lustre-MDT0001 [ 7985.884724] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7985.899119] Lustre: Skipped 2 previous similar messages [ 7992.982720] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8004.581852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.598023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.606659] Lustre: Skipped 1 previous similar message [ 8004.623827] Lustre: Skipped 3 previous similar messages [ 8009.703218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.824156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.841125] Lustre: Skipped 6 previous similar messages [ 8017.375230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8017.720500] Lustre: server umount lustre-MDT0000 complete [ 8019.944371] LustreError: 239809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.986629] LustreError: 239809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8022.145412] LustreError: 239795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564002 with bad export cookie 6244669263953862440 [ 8022.168530] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.559669] Lustre: server umount lustre-MDT0001 complete [ 8028.757597] Lustre: server umount lustre-OST0000 complete [ 8042.138475] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:33:40 (1787564020) [ 8056.708422] Lustre: server umount lustre-OST0001 complete [ 8066.140376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8081.914162] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8118.498021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.781973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8118.799894] Lustre: Skipped 2 previous similar messages [ 8118.853703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8119.001159] Lustre: lustre-MDT0000: new disk, initializing [ 8119.109791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8119.150396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8124.738393] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8136.830898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.983347] Lustre: 248120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8137.031346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8137.034929] Lustre: Skipped 1 previous similar message [ 8137.089269] Lustre: lustre-MDT0001: new disk, initializing [ 8137.175388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8137.216763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8137.229785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8141.890639] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8152.112483] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8158.906663] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8166.327983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.526768] Lustre: lustre-OST0000: new disk, initializing [ 8166.537502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8166.551264] Lustre: 250088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8166.663485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.811303] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8173.680667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8173.687352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8173.815429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8183.654557] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8186.341202] LustreError: 248126:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8196.359802] LustreError: 248126:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8196.370480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.407316] LustreError: 248128:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8196.636711] Lustre: lustre-OST0001: new disk, initializing [ 8196.653715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8196.664847] Lustre: 251202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8203.714207] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8203.894774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8203.914506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8204.049116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8206.448490] LustreError: 248128:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8215.344864] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.526198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8219.616452] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.624282] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.631982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8219.634925] Lustre: Skipped 3 previous similar messages [ 8219.660511] Lustre: Skipped 1 previous similar message [ 8224.741100] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8224.750366] Lustre: Skipped 1 previous similar message [ 8225.525762] Lustre: server umount lustre-OST0001 complete [ 8229.877127] LustreError: 250492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.896212] LustreError: 250492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8245.217562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.221174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8245.255672] Lustre: Skipped 1 previous similar message [ 8245.269513] Lustre: Skipped 1 previous similar message [ 8250.866506] Lustre: server umount lustre-OST0000 complete [ 8254.950780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.964260] LustreError: Skipped 1 previous similar message [ 8254.973788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8260.845784] Lustre: server umount lustre-MDT0000 complete [ 8264.770234] LustreError: 248113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564244 with bad export cookie 6244669263953865044 [ 8264.774444] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8264.782498] LustreError: 248113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8271.216263] Lustre: server umount lustre-MDT0001 complete [ 8283.220968] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8286.184676] Key type lgssc unregistered [ 8286.543989] LNet: 253238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.550234] LNetError: 253238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8287.591275] LNet: Removed LNI 192.168.202.123@tcp [ 8288.454624] Key type .llcrypt unregistered [ 8288.457102] Key type ._llcrypt unregistered [ 8332.548857] Key type ._llcrypt registered [ 8332.551412] Key type .llcrypt registered [ 8332.721481] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8334.209388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8334.259222] alg: No test for adler32 (adler32-zlib) [ 8335.432756] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 8335.746512] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8337.375194] Key type lgssc registered [ 8338.462470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8346.999057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8347.024550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.370951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8348.434184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8348.778224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.819623] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8364.748283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.885950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8364.921181] Lustre: 255447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8365.145728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.790937] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8379.298236] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8385.486978] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8392.801270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8392.943561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8393.073207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.487226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8399.753313] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8406.947382] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8460.772440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8460.776360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.788576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.282099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8461.293309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.309542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.330277] Lustre: Skipped 1 previous similar message [ 8464.733989] Lustre: server umount lustre-OST0000 complete [ 8470.319976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8486.415141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.008673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.136270] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8500.780589] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8501.352317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8501.376611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8511.655967] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8511.676706] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8542.692818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8542.692912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8542.696291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8542.708053] LustreError: Skipped 1 previous similar message [ 8542.740639] Lustre: Skipped 1 previous similar message [ 8546.800648] Lustre: server umount lustre-OST0000 complete [ 8552.930225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8552.933647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8552.943880] Lustre: Skipped 2 previous similar messages [ 8557.349432] Lustre: server umount lustre-MDT0000 complete [ 8561.268635] LustreError: 255440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564541 with bad export cookie 9942750997453813994 [ 8561.269847] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.277443] LustreError: 255440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8561.665823] Lustre: server umount lustre-MDT0001 complete [ 8573.158285] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8575.786896] Key type lgssc unregistered [ 8576.068089] LNet: 260881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8576.073163] LNetError: 260881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8576.083891] LNet: Removed LNI 192.168.202.123@tcp [ 8577.033306] Key type .llcrypt unregistered [ 8577.037651] Key type ._llcrypt unregistered [ 8601.874446] Key type ._llcrypt registered [ 8601.881489] Key type .llcrypt registered [ 8602.093636] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8603.321450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8603.386934] alg: No test for adler32 (adler32-zlib) [ 8604.391733] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 8604.655626] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8606.359184] Key type lgssc registered [ 8607.222320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8614.658536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8614.681894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.059932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8619.911117] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8626.660722] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.679904] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8627.872803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.216960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.144908] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8639.331214] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8644.480467] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8652.118576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8652.380945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8658.103178] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8661.625011] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8668.580500] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8694.753585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8694.759056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.781676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.833377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.849373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.869367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8700.506116] Lustre: server umount lustre-OST0000 complete [ 8704.992353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8704.997404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.007111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8705.012203] Lustre: Skipped 1 previous similar message [ 8708.073773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8708.074280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.088628] Lustre: Skipped 1 previous similar message [ 8710.456656] Lustre: server umount lustre-MDT0000 complete [ 8713.189235] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8713.202210] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8713.473771] LustreError: 262279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564693 with bad export cookie 8038963137126243586 [ 8713.479891] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8713.489407] LustreError: 262279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8713.714651] Lustre: server umount lustre-MDT0001 complete [ 8723.862968] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8726.737733] Key type lgssc unregistered [ 8727.213722] LNet: 266605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8727.223594] LNetError: 266605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8728.294985] LNet: Removed LNI 192.168.202.123@tcp [ 8729.398068] Key type .llcrypt unregistered [ 8729.400517] Key type ._llcrypt unregistered [ 8746.234645] Key type ._llcrypt registered [ 8746.237589] Key type .llcrypt registered [ 8746.338378] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8747.243159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8747.317330] alg: No test for adler32 (adler32-zlib) [ 8748.496783] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 8748.680537] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8750.359187] Key type lgssc registered [ 8751.253733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8759.855548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8759.880066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.409622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.150910] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8771.565587] LustreError: 267825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.583172] LustreError: 267825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8774.996060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.418145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8779.723958] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8787.599735] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.539369] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8801.134803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8801.568980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.718526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8807.445528] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8814.886290] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8830.433320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8830.440381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.451580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8832.481352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8832.502193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.527825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8836.613884] Lustre: server umount lustre-OST0000 complete [ 8842.210341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8842.217985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8842.235075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.249814] Lustre: Skipped 2 previous similar messages [ 8846.863180] Lustre: server umount lustre-MDT0000 complete [ 8848.869358] LustreError: 267826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.897715] LustreError: 267826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8851.024864] LustreError: 270751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564831 with bad export cookie 9310313229323436485 [ 8851.039272] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8851.460700] Lustre: server umount lustre-MDT0001 complete [ 8866.607480] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8869.725785] Key type lgssc unregistered [ 8870.019891] LNet: 271510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8870.024625] LNetError: 271510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8870.038894] LNet: Removed LNI 192.168.202.123@tcp [ 8870.910325] Key type .llcrypt unregistered [ 8870.912828] Key type ._llcrypt unregistered [ 8897.604871] Key type ._llcrypt registered [ 8897.607137] Key type .llcrypt registered [ 8897.715740] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8899.055120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8899.102676] alg: No test for adler32 (adler32-zlib) [ 8900.166638] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 8900.364923] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8902.008454] Key type lgssc registered [ 8903.349954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8911.286054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8911.307884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.779354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.714686] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8922.103463] LustreError: 272928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.126249] LustreError: 272928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8926.803095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.269795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8932.806569] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8942.530478] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8949.541242] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8958.944654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8959.629412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.893887] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8975.483646] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8982.140877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9008.096809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.097619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.110126] LustreError: Skipped 1 previous similar message [ 9008.115142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.123253] Lustre: Skipped 1 previous similar message [ 9010.943989] Lustre: server umount lustre-OST0000 complete [ 9018.340807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.342101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.360021] Lustre: Skipped 1 previous similar message [ 9018.371115] Lustre: Skipped 2 previous similar messages [ 9021.099265] Lustre: server umount lustre-MDT0000 complete [ 9023.461458] LustreError: 273874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.474536] LustreError: 273874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9024.581538] LustreError: 272910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565004 with bad export cookie 11528153715823718730 [ 9024.595982] LustreError: 272910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9024.602614] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.906820] Lustre: server umount lustre-MDT0001 complete [ 9038.107547] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9041.441637] Key type lgssc unregistered [ 9041.773443] LNet: 277239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9041.779958] LNetError: 277239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9041.793860] LNet: Removed LNI 192.168.202.123@tcp [ 9042.767662] Key type .llcrypt unregistered [ 9042.769684] Key type ._llcrypt unregistered [ 9061.454871] Key type ._llcrypt registered [ 9061.457084] Key type .llcrypt registered [ 9061.555376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9062.734098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9062.901838] alg: No test for adler32 (adler32-zlib) [ 9064.030244] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 9064.318221] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9066.063192] Key type lgssc registered [ 9066.976867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9074.399787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9074.415048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.958368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.451511] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9086.439114] LustreError: 278476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9086.464342] LustreError: 278476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9090.042452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.413492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9094.877482] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9103.546477] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.368233] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9118.423786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.882460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.957929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9126.014674] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9134.282738] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9150.953109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9150.965979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.993977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9151.001116] Lustre: Skipped 1 previous similar message [ 9154.677714] Lustre: server umount lustre-OST0000 complete [ 9161.187953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.191915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9161.215544] Lustre: Skipped 2 previous similar messages [ 9161.255694] Lustre: Skipped 1 previous similar message [ 9165.126332] Lustre: server umount lustre-MDT0000 complete [ 9166.310053] LustreError: 278478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.324631] LustreError: 278478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9169.457426] LustreError: 278442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565149 with bad export cookie 13403291037851409614 [ 9169.466395] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9169.469762] LustreError: 278442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9169.835518] Lustre: server umount lustre-MDT0001 complete [ 9184.859530] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9188.279591] Key type lgssc unregistered [ 9188.569422] LNet: 282182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9188.578937] LNetError: 282182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9188.603534] LNet: Removed LNI 192.168.202.123@tcp [ 9189.363823] Key type .llcrypt unregistered [ 9189.367917] Key type ._llcrypt unregistered [ 9201.973261] Key type ._llcrypt registered [ 9201.977904] Key type .llcrypt registered [ 9202.103900] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9222.281529] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9242.537850] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9243.458766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9243.539351] alg: No test for adler32 (adler32-zlib) [ 9244.636745] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 9244.858858] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9246.535196] Key type lgssc registered [ 9247.945842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9289.766089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9289.817127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.162925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9291.189226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9291.299219] Lustre: lustre-MDT0000: new disk, initializing [ 9291.470138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.493898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9297.048421] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9308.641433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.783839] Lustre: 285737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9308.847199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9308.859550] Lustre: Skipped 1 previous similar message [ 9308.989709] Lustre: lustre-MDT0001: new disk, initializing [ 9309.130387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9309.174479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9309.195194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9315.381334] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9326.540455] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9333.931795] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9341.917965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.269707] Lustre: lustre-OST0000: new disk, initializing [ 9342.276529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9342.287921] Lustre: 287703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9342.391347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9349.201762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9349.219816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9349.348064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9349.499706] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9360.542385] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9366.347165] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9366.613156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9372.276684] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9372.510967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9375.205958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.215314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9375.220653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9375.251397] Lustre: Skipped 1 previous similar message [ 9380.332269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9380.342303] Lustre: Skipped 1 previous similar message [ 9381.483857] Lustre: server umount lustre-OST0000 complete [ 9390.563818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.576706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.577736] Lustre: Skipped 2 previous similar messages [ 9390.600299] Lustre: Skipped 1 previous similar message [ 9392.525861] Lustre: server umount lustre-MDT0000 complete [ 9398.123748] LustreError: 285728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565378 with bad export cookie 15409332750612522177 [ 9398.132498] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9398.134441] LustreError: 285728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9398.433668] Lustre: server umount lustre-MDT0001 complete [ 9411.100678] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:56:29 (1787565389) [ 9419.282151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9419.935243] LustreError: 290118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.952992] LustreError: 290118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9420.007799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.376572] LustreError: 290119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.058974] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9430.502449] LustreError: 290118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.644291] LustreError: 290119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.852195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9439.559708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.791655] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9456.437206] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9462.136648] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9470.524341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9470.867729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9478.097234] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9487.113042] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9504.737802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9504.748265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.773532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9506.784287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9506.790950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.880726] Lustre: server umount lustre-OST0000 complete [ 9511.398196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9511.417651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.437445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9517.652635] Lustre: server umount lustre-MDT0000 complete [ 9521.128924] LustreError: 290113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.147662] LustreError: 290113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9521.785542] LustreError: 290099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565501 with bad export cookie 15409332750612523059 [ 9521.822522] LustreError: 290099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9521.840618] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9522.430284] Lustre: server umount lustre-MDT0001 complete [ 9539.094868] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9542.667408] Key type lgssc unregistered [ 9542.919848] LNet: 293683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9542.932820] LNetError: 293683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9542.947751] LNet: Removed LNI 192.168.202.123@tcp [ 9543.764288] Key type .llcrypt unregistered [ 9543.774698] Key type ._llcrypt unregistered [ 9575.253304] Key type ._llcrypt registered [ 9575.255880] Key type .llcrypt registered [ 9575.377040] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9593.598429] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9594.799683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9594.948739] alg: No test for adler32 (adler32-zlib) [ 9596.040779] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 9596.363059] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9598.047165] Key type lgssc registered [ 9598.942417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9656.113069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9656.163829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.559598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9657.626455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9657.772664] Lustre: lustre-MDT0000: new disk, initializing [ 9657.875728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.904702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9663.177714] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9674.752044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.941519] Lustre: 297510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9675.057763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9675.062952] Lustre: Skipped 1 previous similar message [ 9675.130765] Lustre: lustre-MDT0001: new disk, initializing [ 9675.207248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9675.241661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9675.261280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9680.705695] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9692.446996] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9699.612297] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9706.759414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9707.133826] Lustre: lustre-OST2710: new disk, initializing [ 9707.137773] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9707.150396] Lustre: 299493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9707.248648] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9714.643089] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9715.815617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9715.832261] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9715.977617] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9724.694935] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9731.878682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9732.123891] Lustre: lustre-OST03e8: new disk, initializing [ 9732.128272] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9732.134567] Lustre: 300556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9732.328298] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9738.352481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9738.365352] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9738.479887] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9738.916211] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9747.494591] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9755.938448] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9756.231517] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9761.397391] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9761.716329] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9766.277588] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9766.496640] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9771.810092] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9772.004290] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9777.636102] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.639838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9777.642659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9777.656919] Lustre: Skipped 1 previous similar message [ 9777.894313] Lustre: server umount lustre-MDT0000 complete [ 9782.265933] LustreError: 297505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565762 with bad export cookie 9076955364945556379 [ 9782.285785] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9782.755556] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.776724] Lustre: Skipped 3 previous similar messages [ 9782.913319] Lustre: server umount lustre-MDT0001 complete [ 9788.059507] Lustre: server umount lustre-OST2710 complete [ 9793.033667] Lustre: server umount lustre-OST03e8 complete [ 9812.764986] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9828.111059] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9869.748743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.137468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9870.170883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9870.340149] Lustre: lustre-MDT0000: new disk, initializing [ 9870.489869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.513682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9875.580341] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9887.068949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.194749] Lustre: 305824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9887.285579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9887.304503] Lustre: Skipped 1 previous similar message [ 9887.417260] Lustre: lustre-MDT0001: new disk, initializing [ 9887.534947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9887.632524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9887.655245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9893.018613] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9902.896126] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9909.460697] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9917.621802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9917.909161] Lustre: lustre-OST0000: new disk, initializing [ 9917.920307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9917.929549] Lustre: 307790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9918.033815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.076346] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9928.302186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9928.317437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9928.469311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9934.381270] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9939.388297] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9939.588239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9944.742455] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9944.920653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9949.152673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9949.169717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.179314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9949.194346] Lustre: Skipped 1 previous similar message [ 9949.203922] Lustre: Skipped 4 previous similar messages [ 9953.164305] Lustre: server umount lustre-OST0000 complete [ 9959.394384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9959.401887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9959.418161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9959.452121] LustreError: Skipped 1 previous similar message [ 9964.580344] Lustre: server umount lustre-MDT0000 complete [ 9969.633530] LustreError: 305829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.669613] LustreError: 305829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9970.238703] LustreError: 305816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565950 with bad export cookie 9076955364945558059 [ 9970.245670] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.267240] LustreError: 305816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9971.025787] Lustre: server umount lustre-MDT0001 complete [ 9982.789201] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:06:00 (1787565960) [ 9985.757570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9988.706979] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:06:06 (1787565966) [ 9993.139370] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10028.836280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.312643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10029.342051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10029.562807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10034.226460] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10044.832639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10044.915106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10044.939716] Lustre: 311651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10050.225917] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10072.497426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.816579] Lustre: Denying initial registration attempt from nid 192.168.202.123@tcp, specified as failover [10072.831738] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10072.846445] LustreError: 313104:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [10072.854922] LustreError: 313104:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10072.858860] LustreError: 313104:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10072.942449] Lustre: server umount lustre-OST0000 complete [10072.945062] LustreError: 313104:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10081.248466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10081.257746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.274409] Lustre: Skipped 1 previous similar message [10081.277572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.287270] Lustre: Skipped 1 previous similar message [10086.883584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10086.895284] Lustre: Skipped 2 previous similar messages [10095.071131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10095.255814] Lustre: server umount lustre-MDT0000 complete [10097.124995] LustreError: 311659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.159754] LustreError: 311659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10099.290394] LustreError: 311645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566079 with bad export cookie 9076955364945558941 [10099.298641] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.303865] LustreError: 311645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10099.730178] Lustre: server umount lustre-MDT0001 complete [10111.702033] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [10114.896943] Key type lgssc unregistered [10115.183293] LNet: 314232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10115.191601] LNetError: 314232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10115.209718] LNet: Removed LNI 192.168.202.123@tcp [10116.031973] Key type .llcrypt unregistered [10116.034147] Key type ._llcrypt unregistered [10129.073417] Key type ._llcrypt registered [10129.076361] Key type .llcrypt registered [10129.212761] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10130.634599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10130.707390] alg: No test for adler32 (adler32-zlib) [10131.779417] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [10132.019136] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [10133.743214] Key type lgssc registered [10135.042069] Lustre: Echo OBD driver; http://www.lustre.org/ [10187.581763] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10197.793432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10197.805440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.060691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10199.085989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10199.228107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10204.439384] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10214.215169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.350827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10214.401940] Lustre: 316997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10214.778857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10220.252792] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10229.245420] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10236.550804] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10257.136220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.349604] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10257.530424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10264.298634] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10272.586786] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10277.861770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.867933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.885094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10277.890633] Lustre: Skipped 1 previous similar message [10277.914791] Lustre: Skipped 1 previous similar message [10281.195701] Lustre: server umount lustre-OST0000 complete [10286.560329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10286.572963] LustreError: Skipped 1 previous similar message [10286.575199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.608438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10288.098327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10288.104938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.131823] Lustre: Skipped 1 previous similar message [10293.578100] Lustre: server umount lustre-MDT0000 complete [10297.765210] LustreError: 317750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566277 with bad export cookie 7694480577752685342 [10297.780902] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10297.790852] LustreError: 317750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10298.604478] Lustre: server umount lustre-MDT0001 complete [10310.248706] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [10313.754663] Key type lgssc unregistered [10314.324641] LNet: 320578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10314.333416] LNetError: 320578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10314.354869] LNet: Removed LNI 192.168.202.123@tcp [10315.290117] Key type .llcrypt unregistered [10315.294079] Key type ._llcrypt unregistered [10344.166567] Key type ._llcrypt registered [10344.178075] Key type .llcrypt registered [10344.375661] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10346.026528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10346.152539] alg: No test for adler32 (adler32-zlib) [10347.358611] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [10347.601035] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [10349.303191] Key type lgssc registered [10350.441354] Lustre: Echo OBD driver; http://www.lustre.org/ [10359.571541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10359.597187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.147559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10366.153738] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10371.572241] LustreError: 321998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.605527] LustreError: 321998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10375.636091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10376.162818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10381.379666] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10390.154735] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10397.262235] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10406.640119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10406.900047] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10407.036146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10413.975586] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10422.902991] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10436.218510] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [10437.090528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10437.122151] Lustre: Skipped 2 previous similar messages [10439.294544] Lustre: server umount lustre-MDT0000 complete [10441.300196] LustreError: 323189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.236364] LustreError: 321981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566424 with bad export cookie 7670580562644190397 [10444.249619] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10444.255413] LustreError: 321981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10446.437154] LustreError: 321994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.470071] LustreError: 321994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10446.482737] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.23@tcp (stopping) [10446.512074] Lustre: Skipped 3 previous similar messages [10447.337081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.377454] LustreError: 321994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10449.378914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10449.391970] LustreError: 321994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10449.405134] Lustre: Skipped 1 previous similar message [10450.721424] Lustre: server umount lustre-MDT0001 complete [10460.537695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10472.212375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.730244] LustreError: 323830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.855133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.996310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10495.332529] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10499.042382] LustreError: 325595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.068237] LustreError: 325595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10504.170365] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10504.649032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.991407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10509.002114] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10510.318307] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10517.582295] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.718698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10525.408272] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.769586] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10535.907915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.914757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10535.915879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10535.919191] Lustre: Skipped 1 previous similar message [10546.145159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10546.162514] Lustre: Skipped 4 previous similar messages [10548.191476] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10548.799106] Lustre: server umount lustre-OST0000 complete [10556.384876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10560.286719] Lustre: server umount lustre-MDT0000 complete [10561.521284] LustreError: 325600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.548136] LustreError: 325600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10565.216654] LustreError: 324785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566545 with bad export cookie 7670580562644197978 [10565.232803] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.648164] Lustre: server umount lustre-MDT0001 complete [10579.856617] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [10582.738407] Key type lgssc unregistered [10583.095321] LNet: 328273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10583.102307] LNetError: 328273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10583.118666] LNet: Removed LNI 192.168.202.123@tcp [10584.371370] Key type .llcrypt unregistered [10584.382651] Key type ._llcrypt unregistered [10614.354965] Key type ._llcrypt registered [10614.358323] Key type .llcrypt registered [10614.467332] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10615.861789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10615.974123] alg: No test for adler32 (adler32-zlib) [10617.202712] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [10617.470868] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [10619.199241] Key type lgssc registered [10620.868804] Lustre: Echo OBD driver; http://www.lustre.org/ [10629.167283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10629.186040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.901978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10636.461441] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10641.397967] LustreError: 329693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.426559] LustreError: 329693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10646.388529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10646.497771] LustreError: 329692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.057721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10653.019608] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10663.758701] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10674.022456] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10677.730616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10677.743506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10677.777091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10681.824374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10681.839635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.945594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.963637] Lustre: Skipped 2 previous similar messages [10692.063122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10692.066496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10692.086865] Lustre: Skipped 1 previous similar message [10694.260512] Lustre: server umount lustre-MDT0000 complete [10697.184940] LustreError: 329692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.197543] LustreError: 329692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10699.272518] LustreError: 329674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566679 with bad export cookie 15852029894508262488 [10699.300654] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10699.712087] Lustre: server umount lustre-MDT0001 complete [10716.753629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10717.178690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10717.201541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10717.345374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10722.684528] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10731.722607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.811616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10731.836407] Lustre: 332286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10732.056436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10736.783327] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10745.947321] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10753.483741] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10764.721696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.164950] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10765.187600] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10765.194946] LustreError: 334172:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10765.205932] LustreError: 334172:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10765.223123] LustreError: 334172:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10765.341764] Lustre: server umount lustre-OST0000 complete [10765.346570] LustreError: 334172:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10777.070190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.332962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10777.594547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10785.218129] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10786.020599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10795.310676] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10806.025155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10806.187993] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10806.295784] Lustre: lustre-OST0001: new disk, initializing [10806.303725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10806.310106] Lustre: 335716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10806.424784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10813.155665] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10815.689519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10815.714542] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10815.904177] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10822.280730] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10826.208089] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10826.217786] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.240885] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10832.175638] Lustre: server umount lustre-OST0001 complete [10836.449576] LustreError: 334641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.480290] LustreError: 334641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10836.961754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10836.978862] LustreError: Skipped 1 previous similar message [10836.986387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10837.006340] Lustre: Skipped 1 previous similar message [10837.016780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10837.019923] Lustre: Skipped 3 previous similar messages [10841.615598] LustreError: 334641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.645919] LustreError: 334641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10851.299988] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10851.569990] Lustre: server umount lustre-OST0000 complete [10857.338974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.358546] Lustre: Skipped 1 previous similar message [10857.369257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10857.382715] Lustre: Skipped 6 previous similar messages [10863.775837] Lustre: server umount lustre-MDT0000 complete [10867.559649] LustreError: 334656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566847 with bad export cookie 15852029894508263041 [10867.575370] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10867.580055] LustreError: 334656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10868.165167] Lustre: server umount lustre-MDT0001 complete [10881.460979] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [10884.671515] Key type lgssc unregistered [10885.037948] LNet: 337680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10885.061699] LNetError: 337680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10886.131520] LNet: Removed LNI 192.168.202.123@tcp [10887.131110] Key type .llcrypt unregistered [10887.132972] Key type ._llcrypt unregistered [10975.227250] Key type ._llcrypt registered [10975.229730] Key type .llcrypt registered [10975.424394] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [10993.496537] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10994.812814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10994.983086] alg: No test for adler32 (adler32-zlib) [10996.220988] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [10996.500230] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [10998.176669] Key type lgssc registered [10999.529495] Lustre: Echo OBD driver; http://www.lustre.org/ [11046.877757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11046.922203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11048.233405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11048.261138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11048.383278] Lustre: lustre-MDT0000: new disk, initializing [11048.483553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11048.521138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11053.564456] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11065.692110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.848806] Lustre: 342620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11065.905331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11065.910414] Lustre: Skipped 1 previous similar message [11065.978889] Lustre: lustre-MDT0001: new disk, initializing [11066.099785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.141482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11066.172898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11070.994207] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11081.292990] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11088.547674] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11096.814169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.233390] Lustre: lustre-OST0000: new disk, initializing [11097.242639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11097.260234] Lustre: 344592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11097.356439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11103.867096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11103.881461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11103.988288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11104.923594] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11116.117643] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.866928] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11122.092456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11127.965608] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11128.251331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11134.944429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11134.950253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.965961] LustreError: Skipped 1 previous similar message [11134.969401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11134.992945] Lustre: Skipped 1 previous similar message [11140.067911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.081070] Lustre: Skipped 2 previous similar messages [11145.184440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11145.196647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.502925] Lustre: server umount lustre-OST0000 complete [11153.375929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11153.383557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11153.398577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11153.413828] Lustre: Skipped 1 previous similar message [11155.431871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11157.554866] Lustre: server umount lustre-MDT0000 complete [11162.006599] LustreError: 342612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567142 with bad export cookie 9686776581814068484 [11162.016678] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.047898] LustreError: 342612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11162.632771] Lustre: server umount lustre-MDT0001 complete [11171.992517] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:25:50 (1787567150) [11183.503911] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:26:01 (1787567161) [11198.204345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11198.856633] LustreError: 347350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.874978] LustreError: 347350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11198.938689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11204.077273] LustreError: 347351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.906554] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11209.184838] LustreError: 347350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.307526] LustreError: 347351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.495857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.860358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11221.424885] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11231.153314] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11238.900768] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11246.654342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.025426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.456335] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11262.678909] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11281.578986] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11283.680740] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11285.709897] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11287.895723] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11290.175284] Lustre: Failing over lustre-MDT0000 [11292.540911] Lustre: server umount lustre-MDT0000 complete [11293.665134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11293.679620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.706515] LustreError: 347351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.544835] LustreError: 350262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.590890] LustreError: 350262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11311.583236] Lustre: 340843:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787567275/real 1787567275] req@00000000389b4e48 x1874399829507584/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787567291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11311.612489] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11312.482156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11321.825506] LustreError: 347350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.863293] LustreError: 347350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [11322.301467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11322.383991] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11325.014744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11327.515228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11327.594460] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11327.656193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11329.010476] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11342.765691] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11344.775153] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11349.755267] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11352.366191] Lustre: Failing over lustre-MDT0000 [11352.633956] Lustre: server umount lustre-MDT0000 complete [11353.057073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.093499] Lustre: Skipped 4 previous similar messages [11355.782816] LustreError: 347345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.811619] LustreError: 347345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11369.439670] Lustre: 340843:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787567333/real 1787567333] req@00000000a2e344f1 x1874399829531136/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787567349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11369.514549] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.701313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.096105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11380.153578] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11381.852845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11385.325949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11385.337817] Lustre: Skipped 2 previous similar messages [11385.402255] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11385.450105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11385.726425] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11396.969734] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11399.112602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11410.913232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.917050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11410.922655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.922664] Lustre: Skipped 2 previous similar messages [11410.948057] Lustre: Skipped 1 previous similar message [11412.888649] Lustre: server umount lustre-MDT0000 complete [11416.850697] LustreError: 347331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567396 with bad export cookie 9686776581814071200 [11416.856144] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11416.864097] LustreError: 347331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11417.314696] Lustre: server umount lustre-MDT0001 complete [11431.372717] Lustre: server umount lustre-OST0000 complete [11448.655151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.212520] LustreError: 353719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.235772] LustreError: 353719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [11449.302459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11454.513863] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11463.708596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.864394] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11476.446877] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11483.291652] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11491.180638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11494.526701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11498.902919] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11506.935580] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11520.480401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11520.483785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.493257] LustreError: Skipped 1 previous similar message [11520.494215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.494220] Lustre: Skipped 2 previous similar messages [11520.519630] Lustre: Skipped 2 previous similar messages [11531.231118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11531.423282] Lustre: server umount lustre-OST0000 complete [11540.974458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11540.981516] Lustre: Skipped 8 previous similar messages [11541.612455] Lustre: server umount lustre-MDT0000 complete [11545.878460] LustreError: 353700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567526 with bad export cookie 9686776581814071949 [11545.880994] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11545.886112] LustreError: 353700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11546.220367] Lustre: server umount lustre-MDT0001 complete [11559.646581] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11562.805197] Key type lgssc unregistered [11563.253680] LNet: 357256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11563.274049] LNetError: 357256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11563.325217] LNet: Removed LNI 192.168.202.123@tcp [11564.557852] Key type .llcrypt unregistered [11564.559997] Key type ._llcrypt unregistered [11596.522431] Key type ._llcrypt registered [11596.526101] Key type .llcrypt registered [11596.723894] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11598.042940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11598.056911] alg: No test for adler32 (adler32-zlib) [11599.177751] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [11599.416413] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [11601.095196] Key type lgssc registered [11602.270645] Lustre: Echo OBD driver; http://www.lustre.org/ [11612.993788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11613.012341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11614.502123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11619.474782] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11623.914419] LustreError: 358777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11623.928405] LustreError: 358777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11629.025582] LustreError: 358776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.354604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.900656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11635.014262] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11643.988634] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11650.867428] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11657.285089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.686556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11659.774875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11663.806911] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11671.245536] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.410732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11686.432490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.442254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11688.774597] Lustre: server umount lustre-MDT0000 complete [11689.445840] LustreError: 359773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.455152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.661894] LustreError: 358759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567672 with bad export cookie 11896116351225308503 [11692.678029] LustreError: 358759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11692.679056] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11693.132893] Lustre: server umount lustre-MDT0001 complete [11708.083823] Lustre: server umount lustre-OST0000 complete [11723.613689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.060689] LustreError: 362515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.081020] LustreError: 362515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11724.195618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.380202] LustreError: 362516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.424236] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11739.618866] LustreError: 362515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.642959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.652948] LustreError: 362515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11740.523766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11746.150470] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11753.340112] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11758.927714] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11766.918681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11767.353297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11770.433242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11775.168876] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11784.634665] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11802.095053] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11802.097433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11802.100164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11802.110468] Lustre: Skipped 3 previous similar messages [11804.486715] Lustre: server umount lustre-MDT0000 complete [11807.207555] LustreError: 362512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.231611] LustreError: 362512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11809.148629] LustreError: 362495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567789 with bad export cookie 11896116351225309616 [11809.161874] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11809.171130] LustreError: 362495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11809.836768] Lustre: server umount lustre-MDT0001 complete [11824.935307] Lustre: server umount lustre-OST0000 complete [11840.835473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11841.178131] LustreError: 366425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11841.245889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11846.255302] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11855.936695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.915894] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11868.960261] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11875.638274] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11883.287613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11883.550662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.556124] Lustre: Skipped 1 previous similar message [11886.643395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11890.210969] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11898.562562] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11906.019906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.026881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11906.027486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11906.027491] Lustre: Skipped 2 previous similar messages [11906.028423] Lustre: Skipped 1 previous similar message [11910.509985] Lustre: server umount lustre-OST0000 complete [11916.257076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.259915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11916.284321] Lustre: Skipped 1 previous similar message [11916.312876] Lustre: Skipped 2 previous similar messages [11920.635335] Lustre: server umount lustre-MDT0000 complete [11921.378321] LustreError: 366425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.404359] LustreError: 366425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11924.563597] LustreError: 366405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567904 with bad export cookie 11896116351225310638 [11924.565116] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11924.583783] LustreError: 366405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11924.935374] Lustre: server umount lustre-MDT0001 complete [11938.337886] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11941.662930] Key type lgssc unregistered [11941.955754] LNet: 369962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11941.967586] LNetError: 369962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11941.994328] LNet: Removed LNI 192.168.202.123@tcp [11942.905725] Key type .llcrypt unregistered [11942.907748] Key type ._llcrypt unregistered [11971.484268] Key type ._llcrypt registered [11971.485831] Key type .llcrypt registered [11971.563951] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11972.678246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11972.719699] alg: No test for adler32 (adler32-zlib) [11973.769267] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [11974.009030] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [11975.672627] Key type lgssc registered [11976.970465] Lustre: Echo OBD driver; http://www.lustre.org/ [11985.504942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11985.519365] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.524507] LustreError: 371406:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11985.541808] LustreError: 371406:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11985.549452] LustreError: 371406:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11985.558031] LustreError: 371406:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11985.568264] LustreError: 371406:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11994.644184] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.649745] LustreError: 371805:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11994.677764] LustreError: 371805:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11994.682666] LustreError: 371805:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11994.687672] LustreError: 371805:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11994.695211] LustreError: 371805:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12017.461958] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12021.485556] Key type lgssc unregistered [12021.771513] LNet: 372811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12021.776544] LNetError: 372811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12021.816722] LNet: Removed LNI 192.168.202.123@tcp [12023.067935] Key type .llcrypt unregistered [12023.074419] Key type ._llcrypt unregistered [12046.570446] Key type ._llcrypt registered [12046.586321] Key type .llcrypt registered [12046.691538] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12066.007571] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12067.644993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12067.665593] alg: No test for adler32 (adler32-zlib) [12068.717160] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [12069.042179] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12070.761367] Key type lgssc registered [12071.974180] Lustre: Echo OBD driver; http://www.lustre.org/ [12111.556102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12111.594962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.958467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12112.984903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12113.109656] Lustre: lustre-MDT0000: new disk, initializing [12113.243490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12113.266901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12118.312340] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12132.241164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12132.391480] Lustre: 375970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12132.446453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12132.450189] Lustre: Skipped 1 previous similar message [12132.518787] Lustre: lustre-MDT0001: new disk, initializing [12132.606633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12132.637664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12132.658208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12137.486629] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12147.754737] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12153.659951] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12161.124951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12161.330271] Lustre: lustre-OST0000: new disk, initializing [12161.333725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12161.339478] Lustre: 377938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12161.421926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.842392] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12170.310760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12170.320369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12170.428688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12179.302497] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12184.504137] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12184.868443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12190.267314] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12190.508121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12196.328374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.332163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12196.335226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.359330] LustreError: Skipped 1 previous similar message [12201.452142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12201.467495] Lustre: Skipped 2 previous similar messages [12206.561313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.576644] Lustre: Skipped 1 previous similar message [12207.071165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12207.256931] Lustre: server umount lustre-OST0000 complete [12211.686965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.701462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12211.716342] Lustre: Skipped 2 previous similar messages [12216.804868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12216.810960] Lustre: Skipped 2 previous similar messages [12217.683117] Lustre: server umount lustre-MDT0000 complete [12222.323573] LustreError: 376775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568202 with bad export cookie 10884561912076692422 [12222.336411] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12222.341634] LustreError: 376775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12222.783384] Lustre: server umount lustre-MDT0001 complete [12232.970439] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12234.938318] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:43:33 (1787568213) [12244.087784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.069174] LustreError: 380399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.101164] LustreError: 380399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12245.143647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12250.608952] LustreError: 380400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.661704] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12255.728762] LustreError: 380399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.833368] LustreError: 380400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.135235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12262.405311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12266.446510] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12273.670188] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12280.057808] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12287.282081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12287.506538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12294.729334] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12305.727474] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12314.332692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12314.496388] Lustre: lustre-OST0001: new disk, initializing [12314.499703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12314.515909] Lustre: 383235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12314.621707] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12321.553640] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12323.964813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12323.982704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12324.124296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12333.716755] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12339.683436] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.699885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12339.714677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12339.721750] Lustre: Skipped 1 previous similar message [12345.510434] Lustre: server umount lustre-OST0001 complete [12346.975620] LustreError: 382222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.061832] LustreError: 382222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.079344] LustreError: 382222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12367.328617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12367.341204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.373198] Lustre: Skipped 1 previous similar message [12367.392603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12367.405589] Lustre: Skipped 3 previous similar messages [12373.125346] Lustre: server umount lustre-OST0000 complete [12380.128946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12380.137692] LustreError: Skipped 1 previous similar message [12380.145060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.158622] Lustre: Skipped 1 previous similar message [12384.073381] Lustre: server umount lustre-MDT0000 complete [12385.763437] LustreError: 380400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12385.804534] LustreError: 380400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12388.437230] LustreError: 380379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568368 with bad export cookie 10884561912076693304 [12388.440187] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12388.447294] LustreError: 380379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12395.249421] Lustre: server umount lustre-MDT0001 complete [12413.206929] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12416.985572] Key type lgssc unregistered [12417.309930] LNet: 385253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12417.318695] LNetError: 385253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12417.355566] LNet: Removed LNI 192.168.202.123@tcp [12418.473796] Key type .llcrypt unregistered [12418.476606] Key type ._llcrypt unregistered [12474.152926] Key type ._llcrypt registered [12474.161436] Key type .llcrypt registered [12474.354268] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12476.028946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12476.241243] alg: No test for adler32 (adler32-zlib) [12477.448857] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [12477.794370] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12479.495540] Key type lgssc registered [12480.494732] Lustre: Echo OBD driver; http://www.lustre.org/ [12487.708942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12487.724842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12489.075977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12489.109047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12489.294618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12494.200507] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12502.411650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12502.575729] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12502.615847] Lustre: 388054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12502.841237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12507.890525] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12514.982231] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12522.715850] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12528.104260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.115156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12528.124146] Lustre: Skipped 1 previous similar message [12533.222619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12538.343144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12538.350255] Lustre: Skipped 2 previous similar messages [12539.359464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12539.986908] Lustre: server umount lustre-MDT0000 complete [12544.412244] LustreError: 388048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568524 with bad export cookie 4879145853128586701 [12544.416036] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12544.420238] LustreError: 388048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12545.185757] Lustre: server umount lustre-MDT0001 complete [12556.728972] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:48:55 (1787568535) [12563.948856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12564.591501] LustreError: 390571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12564.608854] LustreError: 390571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12564.656614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12569.530939] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12570.087379] LustreError: 390572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12575.200101] LustreError: 390571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12578.341212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12578.621276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12582.591570] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12591.067831] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12597.115593] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12606.272864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12606.439793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12606.625555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12614.083849] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12621.594940] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12629.935878] Lustre: Permanently deactivating lustre-OST0000 [12629.947989] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12640.270834] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12641.268214] LustreError: 393609:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12641.278796] LustreError: 393609:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12641.284392] LustreError: 393609:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12644.382287] LustreError: 393657:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12644.392259] LustreError: 393657:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12652.539726] Lustre: server umount lustre-OST0000 complete [12654.983621] LustreError: 393906:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12654.992578] LustreError: 393906:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12658.147074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12658.154241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12658.165912] Lustre: Skipped 2 previous similar messages [12663.269755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12663.272513] Lustre: Skipped 2 previous similar messages [12666.426423] Lustre: server umount lustre-MDT0000 complete [12668.384735] LustreError: 390567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.396237] LustreError: 390567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12669.493181] LustreError: 390552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568649 with bad export cookie 4879145853128587275 [12669.514404] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12679.756761] Lustre: server umount lustre-MDT0001 complete [12690.701827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.075800] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12699.790464] LustreError: 395065:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12708.082465] Lustre: 395404:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.123@tcp [12710.797979] Lustre: server umount MGS complete [12727.658333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12728.340394] LustreError: 396131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.369268] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12728.373756] Lustre: Skipped 1 previous similar message [12728.466992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12734.194894] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12739.042739] LustreError: 396131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.076351] LustreError: 396131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12745.772675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12746.013535] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12746.110680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12751.348969] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12760.758362] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12768.080797] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12770.917959] Lustre: Permanently reactivating lustre-OST0000 [12770.936378] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12770.940889] Lustre: Skipped 2 previous similar messages [12797.920571] LustreError: 397865:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12807.135914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12807.663270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12814.096373] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12822.632869] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.164057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.176272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12829.188448] Lustre: Skipped 1 previous similar message [12829.220812] Lustre: Skipped 1 previous similar message [12835.187192] Lustre: server umount lustre-OST0000 complete [12839.392667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12839.407872] Lustre: Skipped 1 previous similar message [12844.916779] Lustre: server umount lustre-MDT0000 complete [12849.133065] LustreError: 396111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568829 with bad export cookie 4879145853128588332 [12849.141669] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.145971] LustreError: 396111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12849.384840] Lustre: server umount lustre-MDT0001 complete [12862.536832] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12865.371700] Key type lgssc unregistered [12865.613827] LNet: 400451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12865.619653] LNetError: 400451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12865.641620] LNet: Removed LNI 192.168.202.123@tcp [12866.582156] Key type .llcrypt unregistered [12866.585072] Key type ._llcrypt unregistered [12889.996560] Key type ._llcrypt registered [12890.000577] Key type .llcrypt registered [12890.143753] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12912.649992] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12914.026116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12914.335388] alg: No test for adler32 (adler32-zlib) [12915.481888] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [12915.798028] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12917.439154] Key type lgssc registered [12918.571815] Lustre: Echo OBD driver; http://www.lustre.org/ [12959.321202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12959.356600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12960.764799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12960.814858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12960.982785] Lustre: lustre-MDT0000: new disk, initializing [12961.217758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12961.253859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12966.462882] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12978.691830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.923817] Lustre: 403608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12978.974859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12978.982579] Lustre: Skipped 1 previous similar message [12979.138312] Lustre: lustre-MDT0001: new disk, initializing [12979.276971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12979.335756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12979.356699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12984.326847] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12995.996458] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13004.381094] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13013.202267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13013.607472] Lustre: lustre-OST0000: new disk, initializing [13013.614596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13013.628736] Lustre: 405580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13013.791069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13019.760707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13019.787409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13019.957552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13021.388887] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13033.409381] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13038.813521] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13039.111150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13044.273501] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13044.550953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13050.855496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13050.856379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13050.863432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13050.871129] LustreError: Skipped 1 previous similar message [13050.900628] Lustre: Skipped 1 previous similar message [13052.881909] Lustre: server umount lustre-OST0000 complete [13061.089866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.102552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13061.115696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13061.133368] Lustre: Skipped 1 previous similar message [13066.209066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13066.220484] Lustre: Skipped 1 previous similar message [13071.327313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13071.337145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13071.352265] Lustre: Skipped 1 previous similar message [13071.840677] Lustre: server umount lustre-MDT0000 complete [13076.407961] LustreError: 403600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569056 with bad export cookie 6539464675484908861 [13076.414719] LustreError: 403600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13076.415150] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13076.670678] Lustre: server umount lustre-MDT0001 complete [13088.032201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13090.713382] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:57:49 (1787569069) [13098.752535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13099.209849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13104.000649] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13104.611766] LustreError: 408045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13104.631295] LustreError: 408045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13109.728636] LustreError: 408044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13114.311405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.717605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13119.927716] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13128.204611] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13134.102694] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13141.779304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13142.095691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13149.388438] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13158.790744] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13160.792773] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13163.703979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13170.154027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13170.162150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.165628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.182420] Lustre: Skipped 2 previous similar messages [13175.067270] Lustre: server umount lustre-OST0000 complete [13180.403081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.409946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.422851] Lustre: Skipped 1 previous similar message [13180.442176] Lustre: Skipped 2 previous similar messages [13194.207185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13194.488415] Lustre: server umount lustre-MDT0000 complete [13195.744521] LustreError: 409970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13195.764423] LustreError: 409970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13199.834393] LustreError: 408026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569179 with bad export cookie 6539464675484909743 [13199.845705] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13199.859872] LustreError: 408026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13200.586849] Lustre: server umount lustre-MDT0001 complete [13216.469957] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13219.817289] Key type lgssc unregistered [13220.203252] LNet: 411779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13220.211648] LNetError: 411779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13220.231740] LNet: Removed LNI 192.168.202.123@tcp [13221.645115] Key type .llcrypt unregistered [13221.650967] Key type ._llcrypt unregistered [13249.159631] Key type ._llcrypt registered [13249.161232] Key type .llcrypt registered [13249.307244] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13249.746600] Key type .llcrypt unregistered [13249.748338] Key type ._llcrypt unregistered [13271.637276] Key type ._llcrypt registered [13271.642361] Key type .llcrypt registered [13271.786912] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13273.085214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13273.135185] alg: No test for adler32 (adler32-zlib) [13274.377331] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [13274.675666] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13276.361661] Key type lgssc registered [13277.809683] Lustre: Echo OBD driver; http://www.lustre.org/ [13287.660121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13287.684836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13289.244585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13294.511831] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13299.690724] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13299.729946] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13304.808548] LustreError: 414006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13305.149572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13305.463372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13310.952698] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13310.989427] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13313.361218] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13314.030579] LustreError: 414006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13314.056319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13314.065983] LustreError: 414006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13319.139926] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13319.178897] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13321.785880] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13329.386935] LustreError: 414006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.408691] LustreError: 414006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13330.149725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13330.653238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13336.711664] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13346.286083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.293594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13346.297332] Lustre: Skipped 1 previous similar message [13346.299328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13346.315535] Lustre: Skipped 1 previous similar message [13351.407677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13351.416514] Lustre: Skipped 1 previous similar message [13356.520552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13356.532471] Lustre: Skipped 1 previous similar message [13360.095676] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13360.287661] Lustre: server umount lustre-OST0000 complete [13366.760160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13366.763564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13366.773918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13366.774435] Lustre: Skipped 1 previous similar message [13366.794582] Lustre: Skipped 1 previous similar message [13370.691782] Lustre: server umount lustre-MDT0000 complete [13371.879336] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13371.897711] LustreError: 414007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13374.773541] LustreError: 413987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569354 with bad export cookie 10542245714491121860 [13374.778387] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13374.782360] LustreError: 413987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13375.145408] Lustre: server umount lustre-MDT0001 complete [13386.746252] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13390.238781] Key type lgssc unregistered [13390.566474] LNet: 417161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13390.573729] LNetError: 417161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13390.596416] LNet: Removed LNI 192.168.202.123@tcp [13391.436149] Key type .llcrypt unregistered [13391.439673] Key type ._llcrypt unregistered [13420.000584] Key type ._llcrypt registered [13420.002542] Key type .llcrypt registered [13420.220388] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13421.760081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13421.896142] alg: No test for adler32 (adler32-zlib) [13422.908840] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [13423.122692] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13424.831397] Key type lgssc registered [13426.120650] Lustre: Echo OBD driver; http://www.lustre.org/ [13434.172667] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13434.198673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13449.823426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13454.945580] LustreError: 418552:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.123@tcp: failed processing log, type 4: rc = -110 [13481.635990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13488.715353] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13498.665030] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13507.317154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13508.032190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13508.129793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13513.191244] LustreError: 418577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.214768] LustreError: 418577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13513.365631] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13516.288276] LustreError: 418578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13516.315560] LustreError: 418578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13521.379526] LustreError: 419591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13521.396748] LustreError: 419591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13523.357981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13523.826630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13529.192467] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13538.938113] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13547.100917] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13556.206920] Lustre: server umount lustre-OST0000 complete [13561.836770] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13561.848640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.866461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13564.897096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.897717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13564.925897] Lustre: Skipped 1 previous similar message [13570.021901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13570.024642] Lustre: Skipped 1 previous similar message [13571.195789] Lustre: server umount lustre-MDT0000 complete [13575.135854] Lustre: 417889:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787569539/real 1787569539] req@00000000e5ee5565 x1874402374353408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787569555 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13575.142339] LustreError: 419594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.184238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13575.219652] LustreError: 419594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13576.169793] LustreError: 418558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569556 with bad export cookie 8296722864200232607 [13576.183798] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13576.488522] Lustre: server umount lustre-MDT0001 complete [13590.424661] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13592.908630] Key type lgssc unregistered [13593.317930] LNet: 422123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13593.326446] LNetError: 422123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13593.351710] LNet: Removed LNI 192.168.202.123@tcp [13594.493300] Key type .llcrypt unregistered [13594.495812] Key type ._llcrypt unregistered [13624.361334] Key type ._llcrypt registered [13624.363766] Key type .llcrypt registered [13624.513348] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13625.738404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13625.790434] alg: No test for adler32 (adler32-zlib) [13627.307676] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [13627.588258] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13629.296386] Key type lgssc registered [13630.470142] Lustre: Echo OBD driver; http://www.lustre.org/ [13638.549711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13638.581081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13640.045754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13644.548135] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13650.403375] LustreError: 423541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13650.414427] LustreError: 423541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13653.673798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13654.140239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13659.377162] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13667.194843] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13673.907469] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13681.000605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13681.353834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13685.420313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13688.848554] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13696.296351] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13700.066240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13700.075667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13700.088120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13703.288272] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [13705.198938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13705.205505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13705.222137] Lustre: Skipped 1 previous similar message [13705.233842] Lustre: Skipped 2 previous similar messages [13706.523931] Lustre: server umount lustre-MDT0000 complete [13708.397704] LustreError: 423536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13709.865122] LustreError: 423522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.23@tcp arrived at 1787569689 with bad export cookie 1791332215926708903 [13709.894369] LustreError: 423522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13710.304242] LustreError: 424281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13711.876196] LustreError: 423521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569692 with bad export cookie 1791332215926708014 [13711.879446] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13711.888164] LustreError: 423521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13715.425001] LustreError: 423542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13715.425991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13715.450278] LustreError: 423542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13715.503257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13718.244456] Lustre: server umount lustre-OST0000 complete [13721.587443] LustreError: 423542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.620285] LustreError: 423542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13726.724463] Lustre: server umount lustre-MDT0001 complete [13740.418705] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13743.505195] Key type lgssc unregistered [13743.960451] LNet: 427230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13743.977498] LNetError: 427230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13744.014804] LNet: Removed LNI 192.168.202.123@tcp [13745.285158] Key type .llcrypt unregistered [13745.289257] Key type ._llcrypt unregistered [13776.005698] Key type ._llcrypt registered [13776.009941] Key type .llcrypt registered [13776.157645] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13778.226384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13778.257990] alg: No test for adler32 (adler32-zlib) [13779.537659] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [13779.879873] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13781.703206] Key type lgssc registered [13783.821460] Lustre: Echo OBD driver; http://www.lustre.org/ [13793.546068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13793.590269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13795.316087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13799.300626] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13805.542661] LustreError: 428651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13805.573021] LustreError: 428651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13808.441621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13808.910419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13814.318189] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13822.546443] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13829.408797] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13837.278371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13837.749778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13840.832159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13844.076584] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13852.475628] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13858.280710] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13858.302978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13858.316817] Lustre: Skipped 1 previous similar message [13859.445399] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.23@tcp (stopping) [13862.697865] Lustre: server umount lustre-MDT0001 complete [13863.409910] LustreError: 428652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13863.443966] LustreError: 428652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13864.549416] LustreError: 428648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.515214] LustreError: 428651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.533243] LustreError: 428651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13871.587097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13871.596402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13871.613035] Lustre: Skipped 1 previous similar message [13871.625678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13873.647904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13873.648784] LustreError: 429625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13876.036612] Lustre: server umount lustre-OST0000 complete [13881.336488] Lustre: server umount lustre-MDT0000 complete [13897.430425] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13900.842083] Key type lgssc unregistered [13901.238071] LNet: 432341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13901.253514] LNetError: 432341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13901.282185] LNet: Removed LNI 192.168.202.123@tcp [13902.265464] Key type .llcrypt unregistered [13902.273609] Key type ._llcrypt unregistered [13929.825216] Key type ._llcrypt registered [13929.828476] Key type .llcrypt registered [13929.936317] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [13947.452376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13948.910104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13948.946556] alg: No test for adler32 (adler32-zlib) [13950.088249] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [13950.384820] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13952.087196] Key type lgssc registered [13954.190849] Lustre: Echo OBD driver; http://www.lustre.org/ [13997.296053] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [14016.223205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14016.271688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14017.575328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14017.633468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14017.738985] Lustre: lustre-MDT0000: new disk, initializing [14017.802364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14017.817463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14022.866276] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14040.246222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14040.506457] Lustre: lustre-OST0000: new disk, initializing [14040.510154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14040.514541] Lustre: Skipped 1 previous similar message [14040.519085] Lustre: 437378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14040.598978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14047.661600] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14048.383974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14048.402815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14048.444554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14059.260395] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14078.351031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14078.494674] Lustre: 436361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14078.556673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14078.609211] Lustre: lustre-MDT0001: new disk, initializing [14078.727924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14078.758580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14078.772954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14083.425872] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14093.496275] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14093.763880] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14098.704303] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14098.906682] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14107.620287] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14107.622489] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14107.640933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14107.659763] Lustre: Skipped 1 previous similar message [14112.744297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14112.754564] Lustre: Skipped 2 previous similar messages [14117.884867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14117.889699] Lustre: Skipped 2 previous similar messages [14118.368143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14118.790248] Lustre: server umount lustre-MDT0001 complete [14128.101721] LustreError: 436372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14128.125527] LustreError: 436372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14133.220388] LustreError: 436368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14133.235139] LustreError: 436368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14138.338713] LustreError: 436371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14138.339628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14138.388350] Lustre: Skipped 1 previous similar message [14138.397289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14138.406029] Lustre: Skipped 1 previous similar message [14143.395332] Lustre: server umount lustre-OST0000 complete [14143.468255] LustreError: 436372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14154.798691] Lustre: server umount lustre-MDT0000 complete [14171.708720] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [14175.804984] Key type lgssc unregistered [14176.282453] LNet: 441147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14176.296042] LNetError: 441147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14176.318087] LNet: Removed LNI 192.168.202.123@tcp [14177.307634] Key type .llcrypt unregistered [14177.311259] Key type ._llcrypt unregistered [14227.390119] Key type ._llcrypt registered [14227.397365] Key type .llcrypt registered [14227.584129] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [14227.954760] Key type .llcrypt unregistered [14227.960971] Key type ._llcrypt unregistered [14250.962708] Key type ._llcrypt registered [14250.964588] Key type .llcrypt registered [14251.057915] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [14252.727180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14252.744111] alg: No test for adler32 (adler32-zlib) [14253.988864] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [14254.320035] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [14256.022502] Key type lgssc registered [14257.561821] Lustre: Echo OBD driver; http://www.lustre.org/ [14289.943538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14289.973123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14291.425146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14291.478019] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14291.555792] Lustre: lustre-MDT0000: new disk, initializing [14291.619689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14291.628791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14296.921715] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14309.645256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14309.812960] Lustre: 444530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14309.887144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14309.893547] Lustre: Skipped 1 previous similar message [14309.988270] Lustre: lustre-MDT0001: new disk, initializing [14310.099736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14310.143031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14310.152420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14315.421662] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14326.851234] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14334.425376] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14342.609263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14342.905289] Lustre: lustre-OST0000: new disk, initializing [14342.913619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14342.926661] Lustre: 446500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14343.039143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14349.953333] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14350.413985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14350.430714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14350.483685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14362.353827] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14371.305913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14371.309675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14371.336789] Lustre: Skipped 2 previous similar messages [14371.361385] Lustre: Skipped 2 previous similar messages [14376.419329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14376.784862] Lustre: server umount lustre-MDT0000 complete [14381.897459] LustreError: 444525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570362 with bad export cookie 12368816289025894684 [14381.902801] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14381.906356] LustreError: 444525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14382.477345] Lustre: server umount lustre-MDT0001 complete [14397.114738] Lustre: server umount lustre-OST0000 complete [14411.603281] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:19:50 (1787570390) [14422.610562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14423.086923] LustreError: 448530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14423.099368] LustreError: 448530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14423.204399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14428.641112] LustreError: 448531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14429.449150] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14433.763820] LustreError: 448530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14438.896654] LustreError: 448531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14441.592877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14442.313261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14448.537482] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14457.308910] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14465.245960] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14487.801760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14488.127085] Lustre: Found index 0 for lustre-OST0000, updating log [14488.362830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14493.502790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14496.152335] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14505.432975] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14512.500363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14512.525501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14512.530384] Lustre: Skipped 2 previous similar messages [14512.628121] Lustre: server umount lustre-OST0000 complete [14529.504691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14529.521745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14529.582431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14534.111982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14534.112212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14535.589888] Lustre: server umount lustre-MDT0000 complete [14539.232699] LustreError: 448531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14539.269237] LustreError: 448531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14540.428154] LustreError: 448511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570520 with bad export cookie 12368816289025896070 [14540.452447] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14540.788396] Lustre: server umount lustre-MDT0001 complete [14552.845725] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:22:11 (1787570531) [14555.552331] LNet: 1 local NIs in recovery (showing 1): 192.168.202.123@tcp [15183.170928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15183.669911] LustreError: 454944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15183.787793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15188.366349] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15188.968464] LustreError: 454945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15193.057296] LustreError: 454944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15197.226726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15197.573256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15202.183569] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15211.368218] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15217.925937] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15234.655410] LNet: 1 local NIs in recovery (showing 1): 192.168.202.123@tcp [15238.508021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15238.778625] Lustre: Found index 0 for lustre-OST0000, updating log [15239.019902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15245.062535] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15253.467498] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15256.738978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15256.751963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15256.754665] Lustre: Skipped 1 previous similar message [15269.855148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15270.366512] Lustre: server umount lustre-OST0000 complete [15276.068922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15276.072493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15276.093094] Lustre: Skipped 1 previous similar message [15278.177164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15278.203365] Lustre: Skipped 4 previous similar messages [15281.400833] Lustre: server umount lustre-MDT0000 complete [15282.275352] LustreError: 456761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15282.303660] LustreError: 456761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15285.953947] LustreError: 454926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571266 with bad export cookie 12368816289025897148 [15285.973951] LustreError: 454926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15285.984067] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15286.318983] Lustre: server umount lustre-MDT0001 complete [15294.048093] LNet: 1 local NIs in recovery (showing 1): 192.168.202.123@tcp [15318.384956] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [15328.898582] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [15371.734848] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:35:50 (1787571350) [15392.734272] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [15395.030649] Key type lgssc unregistered [15395.346721] LNet: 461572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15395.354916] LNetError: 461572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15500.836867] LNet: Removed LNI 192.168.202.123@tcp [15501.760257] Key type .llcrypt unregistered [15501.764080] Key type ._llcrypt unregistered [15523.192564] Key type ._llcrypt registered [15523.195533] Key type .llcrypt registered [15523.312941] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [15525.018321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15525.102553] alg: No test for adler32 (adler32-zlib) [15526.212775] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [15526.435940] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [15528.111688] Key type lgssc registered [15529.326282] Lustre: Echo OBD driver; http://www.lustre.org/ [15533.784460] LNet: Added LNI 192.168.202.123@tcp100 [8/256/0/180] [15534.569581] LNet: Added LNI 192.168.202.123@tcp101 [8/256/0/180] [15535.528905] LNet: Added LNI 192.168.202.123@tcp102 [8/256/0/180] [15536.601154] LNet: Added LNI 192.168.202.123@tcp103 [8/256/0/180] [15537.571683] LNet: Added LNI 192.168.202.123@tcp104 [8/256/0/180] [15538.636657] LNet: Added LNI 192.168.202.123@tcp105 [8/256/0/180] [15539.660321] LNet: Added LNI 192.168.202.123@tcp106 [8/256/0/180] [15540.834626] LNet: Added LNI 192.168.202.123@tcp107 [8/256/0/180] [15541.781759] LNet: Added LNI 192.168.202.123@tcp108 [8/256/0/180] [15542.884388] LNet: Added LNI 192.168.202.123@tcp109 [8/256/0/180] [15543.910769] LNet: Added LNI 192.168.202.123@tcp110 [8/256/0/180] [15545.192727] LNet: Added LNI 192.168.202.123@tcp111 [8/256/0/180] [15546.337906] LNet: Added LNI 192.168.202.123@tcp112 [8/256/0/180] [15547.093360] LNet: Added LNI 192.168.202.123@tcp113 [8/256/0/180] [15547.956783] LNet: Added LNI 192.168.202.123@tcp114 [8/256/0/180] [15548.801685] LNet: Added LNI 192.168.202.123@tcp115 [8/256/0/180] [15550.031701] LNet: Added LNI 192.168.202.123@tcp116 [8/256/0/180] [15550.924244] LNet: Added LNI 192.168.202.123@tcp117 [8/256/0/180] [15551.804645] LNet: Added LNI 192.168.202.123@tcp118 [8/256/0/180] [15552.667495] LNet: Added LNI 192.168.202.123@tcp119 [8/256/0/180] [15553.633266] LNet: Added LNI 192.168.202.123@tcp120 [8/256/0/180] [15554.418050] LNet: Added LNI 192.168.202.123@tcp121 [8/256/0/180] [15555.374460] LNet: Added LNI 192.168.202.123@tcp122 [8/256/0/180] [15556.483326] LNet: Added LNI 192.168.202.123@tcp123 [8/256/0/180] [15557.328984] LNet: Added LNI 192.168.202.123@tcp124 [8/256/0/180] [15558.381710] LNet: Added LNI 192.168.202.123@tcp125 [8/256/0/180] [15559.401625] LNet: Added LNI 192.168.202.123@tcp126 [8/256/0/180] [15560.674743] LNet: Added LNI 192.168.202.123@tcp127 [8/256/0/180] [15561.593908] LNet: Added LNI 192.168.202.123@tcp128 [8/256/0/180] [15562.859572] LNet: Added LNI 192.168.202.123@tcp129 [8/256/0/180] [15564.212955] LNet: Added LNI 192.168.202.123@tcp130 [8/256/0/180] [15565.149444] LNet: Added LNI 192.168.202.123@tcp131 [8/256/0/180] [15565.931391] LNet: Added LNI 192.168.202.123@tcp132 [8/256/0/180] [15566.844888] LNet: Added LNI 192.168.202.123@tcp133 [8/256/0/180] [15567.630385] LNet: Added LNI 192.168.202.123@tcp134 [8/256/0/180] [15568.743095] LNet: Added LNI 192.168.202.123@tcp135 [8/256/0/180] [15576.420117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15576.470677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15577.609781] Lustre: 464605:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.123@tcp: can't add NID 192.168.202.123@tcp130: rc = -75 [15577.702047] LNetError: 462063:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15577.727423] LNetError: Protocol error connecting to 192.168.202.123@tcp117 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15577.748501] LNetError: Skipped 2 previous similar messages [15577.892043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15577.923162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15578.037737] Lustre: lustre-MDT0000: new disk, initializing [15578.174817] Lustre: 464662:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.123@tcp: can't add NID 192.168.202.123@tcp130: rc = -75 [15578.186818] Lustre: 464662:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15578.242213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15578.259159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15578.660391] LNet: 3 local NIs in recovery (showing 3): 192.168.202.123@tcp125, 192.168.202.123@tcp126, 192.168.202.123@tcp127 [15578.661404] LNetError: 462062:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15578.691271] LNetError: 462062:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15578.704209] LNetError: Protocol error connecting to 192.168.202.123@tcp105 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15578.731759] LNetError: Skipped 7 previous similar messages [15580.773276] LNetError: 464630:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15580.790152] LNetError: 464630:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15580.815468] LNetError: Protocol error connecting to 192.168.202.123@tcp127 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15583.509672] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15584.869994] LNetError: 462063:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15584.890747] LNetError: Protocol error connecting to 192.168.202.123@tcp127 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15592.033837] LNetError: 464639:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15592.059505] LNetError: Protocol error connecting to 192.168.202.123@tcp127 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15595.278219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15595.434498] Lustre: 464621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15595.521632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15595.525214] Lustre: Skipped 1 previous similar message [15595.607316] Lustre: lustre-MDT0001: new disk, initializing [15595.654359] Lustre: 465425:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.123@tcp: can't add NID 192.168.202.123@tcp130: rc = -75 [15595.667497] Lustre: 465425:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15595.787367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15595.834253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15595.847675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15601.107051] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15608.418726] LNetError: 464633:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15608.429486] LNetError: Protocol error connecting to 192.168.202.123@tcp127 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15611.800359] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15618.273429] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15625.239291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15625.555163] Lustre: lustre-OST0000: new disk, initializing [15625.558279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15625.564350] Lustre: 466597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15625.654420] Lustre: 466602:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.123@tcp: can't add NID 192.168.202.123@tcp130: rc = -75 [15625.660943] Lustre: 466602:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15625.686764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15625.893440] LNetError: 464633:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15625.908757] LNetError: 464633:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15625.919159] LNetError: Protocol error connecting to 192.168.202.123@tcp123 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15625.933610] LNetError: Skipped 10 previous similar messages [15632.439490] Lustre: 466906:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.123@tcp: can't add NID 192.168.202.123@tcp130: rc = -75 [15632.451988] Lustre: 466906:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15632.546811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15632.558590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15632.739213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15633.168419] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15638.111146] LNet: 1 local NIs in recovery (showing 1): 192.168.202.123@tcp127 [15644.521427] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15653.357176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15653.360574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15653.379514] LustreError: Skipped 1 previous similar message [15653.382193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15653.407270] Lustre: Skipped 1 previous similar message [15653.450661] Lustre: Skipped 1 previous similar message [15658.469669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15658.485105] Lustre: Skipped 1 previous similar message [15659.057966] Lustre: server umount lustre-OST0000 complete [15665.824451] LNetError: 462064:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.123 [15665.840781] LNetError: 462064:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [15665.849660] LNetError: Protocol error connecting to 192.168.202.123@tcp105 at host 192.168.202.123:988: is it running a compatible version of Lustre? [15665.859841] LNetError: Skipped 21 previous similar messages [15667.687082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15667.693113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15667.701559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15668.709839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15670.402910] Lustre: server umount lustre-MDT0000 complete [15675.694561] LustreError: 464615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571655 with bad export cookie 126593776249442212 [15675.715953] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15675.736967] LustreError: 464615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15676.294199] Lustre: server umount lustre-MDT0001 complete [15693.588398] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [15696.529608] Key type lgssc unregistered [15696.927623] LNet: 468516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15696.934129] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15696.940037] LNet: Removed LNI 192.168.202.123@tcp [15697.951168] LNet: Removed LNI 192.168.202.123@tcp100 [15697.963370] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15697.969152] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15697.974747] LNet: Removed LNI 192.168.202.123@tcp101 [15699.041994] LNet: Removed LNI 192.168.202.123@tcp102 [15699.047667] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15699.053846] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15699.065151] LNet: Removed LNI 192.168.202.123@tcp103 [15700.128067] LNet: Removed LNI 192.168.202.123@tcp104 [15701.151125] LNet: Removed LNI 192.168.202.123@tcp105 [15701.152773] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15701.164268] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15701.174743] LNet: Removed LNI 192.168.202.123@tcp106 [15702.240075] LNet: Removed LNI 192.168.202.123@tcp107 [15703.263258] LNet: Removed LNI 192.168.202.123@tcp108 [15704.287496] LNet: Removed LNI 192.168.202.123@tcp109 [15705.311484] LNet: Removed LNI 192.168.202.123@tcp110 [15705.320990] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15705.331208] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15705.349322] LNet: Removed LNI 192.168.202.123@tcp111 [15706.399379] LNet: Removed LNI 192.168.202.123@tcp112 [15707.423159] LNet: Removed LNI 192.168.202.123@tcp113 [15708.511213] LNet: Removed LNI 192.168.202.123@tcp114 [15709.535145] LNet: Removed LNI 192.168.202.123@tcp115 [15710.559183] LNet: Removed LNI 192.168.202.123@tcp116 [15711.583527] LNet: Removed LNI 192.168.202.123@tcp117 [15712.607748] LNet: Removed LNI 192.168.202.123@tcp118 [15713.632788] LNet: Removed LNI 192.168.202.123@tcp119 [15713.656873] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15713.667966] LNetError: 468516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15714.719374] LNet: Removed LNI 192.168.202.123@tcp120 [15715.743253] LNet: Removed LNI 192.168.202.123@tcp121 [15716.767197] LNet: Removed LNI 192.168.202.123@tcp122 [15717.791178] LNet: Removed LNI 192.168.202.123@tcp123 [15718.815178] LNet: Removed LNI 192.168.202.123@tcp124 [15719.839381] LNet: Removed LNI 192.168.202.123@tcp125 [15720.863195] LNet: Removed LNI 192.168.202.123@tcp126 [15720.867864] LNet: Removed LNI 192.168.202.123@tcp127 [15721.889431] LNet: Removed LNI 192.168.202.123@tcp128 [15722.911226] LNet: Removed LNI 192.168.202.123@tcp129 [15722.913310] LNet: Removed LNI 192.168.202.123@tcp130 [15722.915232] LNet: Removed LNI 192.168.202.123@tcp131 [15722.917788] LNet: Removed LNI 192.168.202.123@tcp132 [15722.920155] LNet: Removed LNI 192.168.202.123@tcp133 [15722.922440] LNet: Removed LNI 192.168.202.123@tcp134 [15722.930848] LNet: Removed LNI 192.168.202.123@tcp135 [15723.875398] Key type .llcrypt unregistered [15723.877679] Key type ._llcrypt unregistered [15750.012807] Key type ._llcrypt registered [15750.018121] Key type .llcrypt registered [15750.179727] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [15770.652922] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [15771.909784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15772.017627] alg: No test for adler32 (adler32-zlib) [15773.110877] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [15773.374840] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [15775.032630] Key type lgssc registered [15776.348421] Lustre: Echo OBD driver; http://www.lustre.org/ [15822.382225] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:43:20 (1787571800) [15841.641612] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [15844.666684] Key type lgssc unregistered [15844.982214] LNet: 472613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15844.999364] LNetError: 472613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15845.025358] LNet: Removed LNI 192.168.202.123@tcp [15845.861821] Key type .llcrypt unregistered [15845.869054] Key type ._llcrypt unregistered [15865.976494] Key type ._llcrypt registered [15865.980925] Key type .llcrypt registered [15866.096296] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [15867.323778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15867.393651] alg: No test for adler32 (adler32-zlib) [15868.455757] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [15868.685572] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [15870.319179] Key type lgssc registered [15871.457815] Lustre: Echo OBD driver; http://www.lustre.org/ [15882.750165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15882.834958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15884.085386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15884.129177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15884.264052] Lustre: lustre-MDT0000: new disk, initializing [15884.396211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15884.423084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15890.309516] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15902.561530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15902.650831] Lustre: 473824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15902.692124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15902.695467] Lustre: Skipped 1 previous similar message [15902.749375] Lustre: lustre-MDT0001: new disk, initializing [15902.827236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15902.862465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15902.875872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15908.084785] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15918.837166] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15925.728489] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15933.071303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15933.440668] Lustre: lustre-OST0000: new disk, initializing [15933.446750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15933.479550] Lustre: 475794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15933.627165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15939.911182] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [15943.316062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15943.340740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15943.605510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15951.534558] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15956.185473] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing lnet_if_list [15961.797756] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15961.823419] LNet: Added LNI 192.168.202.123@tcp10 [8/256/0/180] [15968.615885] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15968.662850] LNet: Added LNI 192.168.202.123@tcp11 [8/256/0/180] [15973.595540] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15973.625864] LNet: Added LNI 192.168.202.123@tcp12 [8/256/0/180] [15978.818888] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15978.860154] LNet: Added LNI 192.168.202.123@tcp13 [8/256/0/180] [15983.990949] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15984.037528] LNet: Added LNI 192.168.202.123@tcp14 [8/256/0/180] [15989.887995] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15989.931350] LNet: Added LNI 192.168.202.123@tcp15 [8/256/0/180] [15995.318413] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15995.358270] LNet: Added LNI 192.168.202.123@tcp16 [8/256/0/180] [16001.344360] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16001.370444] LNet: Added LNI 192.168.202.123@tcp17 [8/256/0/180] [16008.135472] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16008.186599] LNet: Added LNI 192.168.202.123@tcp18 [8/256/0/180] [16015.954460] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16016.009575] LNet: Added LNI 192.168.202.123@tcp19 [8/256/0/180] [16023.976084] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16024.002800] LNet: Added LNI 192.168.202.123@tcp20 [8/256/0/180] [16030.877866] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16030.929616] LNet: Added LNI 192.168.202.123@tcp21 [8/256/0/180] [16038.754846] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16038.817535] LNet: Added LNI 192.168.202.123@tcp22 [8/256/0/180] [16045.055801] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16045.086587] LNet: Added LNI 192.168.202.123@tcp23 [8/256/0/180] [16051.924471] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16051.945738] LNet: Added LNI 192.168.202.123@tcp24 [8/256/0/180] [16059.231126] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16059.255478] LNet: Added LNI 192.168.202.123@tcp25 [8/256/0/180] [16065.367233] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16065.404314] LNet: Added LNI 192.168.202.123@tcp26 [8/256/0/180] [16070.886628] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16070.911735] LNet: Added LNI 192.168.202.123@tcp27 [8/256/0/180] [16076.482364] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16076.510062] LNet: Added LNI 192.168.202.123@tcp28 [8/256/0/180] [16082.675045] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16082.701611] LNet: Added LNI 192.168.202.123@tcp29 [8/256/0/180] [16091.395276] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16091.429547] LNet: Added LNI 192.168.202.123@tcp30 [8/256/0/180] [16098.669637] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16098.693603] LNet: Added LNI 192.168.202.123@tcp31 [8/256/0/180] [16106.518566] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16106.545558] LNet: Added LNI 192.168.202.123@tcp32 [8/256/0/180] [16114.205934] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16114.267443] LNet: Added LNI 192.168.202.123@tcp33 [8/256/0/180] [16121.510871] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16121.530161] LNet: Added LNI 192.168.202.123@tcp34 [8/256/0/180] [16129.703221] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16129.736130] LNet: Added LNI 192.168.202.123@tcp35 [8/256/0/180] [16138.787847] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16138.820770] LNet: Added LNI 192.168.202.123@tcp36 [8/256/0/180] [16146.452703] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16146.506409] LNet: Added LNI 192.168.202.123@tcp37 [8/256/0/180] [16156.708734] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16156.751691] LNet: Added LNI 192.168.202.123@tcp38 [8/256/0/180] [16166.594588] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16166.639637] LNet: Added LNI 192.168.202.123@tcp39 [8/256/0/180] [16175.817751] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16175.851746] LNet: Added LNI 192.168.202.123@tcp40 [8/256/0/180] [16183.607724] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16183.661051] LNet: Added LNI 192.168.202.123@tcp41 [8/256/0/180] [16192.201300] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16192.230251] LNet: Added LNI 192.168.202.123@tcp42 [8/256/0/180] [16201.480776] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16201.543438] LNet: Added LNI 192.168.202.123@tcp43 [8/256/0/180] [16212.001318] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16212.063141] LNet: Added LNI 192.168.202.123@tcp44 [8/256/0/180] [16221.957760] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16221.995742] LNet: Added LNI 192.168.202.123@tcp45 [8/256/0/180] [16230.264942] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16230.310069] LNet: Added LNI 192.168.202.123@tcp46 [8/256/0/180] [16238.257170] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16238.311665] LNet: Added LNI 192.168.202.123@tcp47 [8/256/0/180] [16246.792965] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16246.840413] LNet: Added LNI 192.168.202.123@tcp48 [8/256/0/180] [16255.224445] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16255.259338] LNet: Added LNI 192.168.202.123@tcp49 [8/256/0/180] [16264.792613] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16264.858288] LNet: Added LNI 192.168.202.123@tcp50 [8/256/0/180] [16291.809668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16291.810853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16291.812506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16291.824019] Lustre: Skipped 1 previous similar message [16293.645667] Lustre: server umount lustre-MDT0000 complete [16297.701763] LustreError: 473819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572277 with bad export cookie 7030268389696597889 [16297.704165] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16297.717460] LustreError: 473819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16298.052812] Lustre: server umount lustre-MDT0001 complete [16318.431194] Lustre: 473099:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787572282/real 1787572282] req@000000009eb512a1 x1874404938275712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787572298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16318.490581] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16318.528074] Lustre: Skipped 1 previous similar message [16320.017631] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [16323.562057] Lustre: 473097:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787572287/real 1787572287] req@00000000c4a60899 x1874404938275840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787572303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16328.672377] Lustre: 473098:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787572292/real 1787572292] req@00000000a940a0aa x1874404938276096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787572308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16331.196934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16333.791153] Lustre: 473100:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787572297/real 1787572297] req@0000000061d528d3 x1874404938276224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787572313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16342.111651] LustreError: 473096:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075f04b9b x1874404938276864/t0(0) o250->MGC192.168.202.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16342.472395] LustreError: 475787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16342.490256] LustreError: 475787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16342.555307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16343.584412] LustreError: 475788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16343.610232] LustreError: 475788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16347.693075] LustreError: 500063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16347.701318] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16347.716383] LustreError: 500063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16348.356671] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16352.736461] LustreError: 475787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16352.757827] LustreError: 475787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16357.867765] LustreError: 500064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16357.886385] LustreError: 500064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16358.410530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16358.869842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16363.994049] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16364.005867] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16372.966191] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16379.879693] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16402.914519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16402.952325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16402.962235] Lustre: Skipped 3 previous similar messages [16408.033132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16413.157257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16413.164156] Lustre: Skipped 1 previous similar message [16415.199347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16415.446945] Lustre: server umount lustre-OST0000 complete [16420.320720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16420.334632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16420.352173] Lustre: Skipped 1 previous similar message [16420.362651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16425.775675] Lustre: server umount lustre-MDT0000 complete [16428.512783] LustreError: 500067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16428.523948] LustreError: 500067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16429.588482] LustreError: 473817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572409 with bad export cookie 7030268389696600640 [16429.596148] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16429.601154] LustreError: 473817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16429.844427] Lustre: server umount lustre-MDT0001 complete [16439.754319] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [16442.495974] Key type lgssc unregistered [16442.845847] LNet: 502870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16442.854268] LNetError: 502870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16443.871381] LNet: Removed LNI 192.168.202.123@tcp [16443.874621] LNetError: 502870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16443.887477] LNet: Removed LNI 192.168.202.123@tcp10 [16443.895864] LNet: Removed LNI 192.168.202.123@tcp11 [16443.908201] LNet: Removed LNI 192.168.202.123@tcp12 [16443.911916] LNet: Removed LNI 192.168.202.123@tcp13 [16443.926063] LNet: Removed LNI 192.168.202.123@tcp14 [16443.937285] LNet: Removed LNI 192.168.202.123@tcp15 [16443.940500] LNet: Removed LNI 192.168.202.123@tcp16 [16443.947312] LNet: Removed LNI 192.168.202.123@tcp17 [16443.951204] LNet: Removed LNI 192.168.202.123@tcp18 [16443.959150] LNet: Removed LNI 192.168.202.123@tcp19 [16443.960761] LNet: Removed LNI 192.168.202.123@tcp20 [16443.968312] LNet: Removed LNI 192.168.202.123@tcp21 [16443.975981] LNet: Removed LNI 192.168.202.123@tcp22 [16443.982095] LNet: Removed LNI 192.168.202.123@tcp23 [16443.984169] LNet: Removed LNI 192.168.202.123@tcp24 [16443.992980] LNet: Removed LNI 192.168.202.123@tcp25 [16443.997059] LNet: Removed LNI 192.168.202.123@tcp26 [16444.002160] LNet: Removed LNI 192.168.202.123@tcp27 [16444.005467] LNet: Removed LNI 192.168.202.123@tcp28 [16444.019037] LNet: Removed LNI 192.168.202.123@tcp29 [16444.022198] LNet: Removed LNI 192.168.202.123@tcp30 [16444.024595] LNet: Removed LNI 192.168.202.123@tcp31 [16444.026936] LNet: Removed LNI 192.168.202.123@tcp32 [16444.029815] LNet: Removed LNI 192.168.202.123@tcp33 [16444.032129] LNet: Removed LNI 192.168.202.123@tcp34 [16444.034465] LNet: Removed LNI 192.168.202.123@tcp35 [16444.037168] LNet: Removed LNI 192.168.202.123@tcp36 [16444.039863] LNet: Removed LNI 192.168.202.123@tcp37 [16444.042119] LNet: Removed LNI 192.168.202.123@tcp38 [16444.044147] LNet: Removed LNI 192.168.202.123@tcp39 [16444.046756] LNet: Removed LNI 192.168.202.123@tcp40 [16444.052183] LNet: Removed LNI 192.168.202.123@tcp41 [16444.055208] LNet: Removed LNI 192.168.202.123@tcp42 [16444.058413] LNet: Removed LNI 192.168.202.123@tcp43 [16444.064608] LNet: Removed LNI 192.168.202.123@tcp44 [16444.067884] LNet: Removed LNI 192.168.202.123@tcp45 [16444.071751] LNet: Removed LNI 192.168.202.123@tcp46 [16444.074504] LNet: Removed LNI 192.168.202.123@tcp47 [16444.079956] LNet: Removed LNI 192.168.202.123@tcp48 [16444.083694] LNet: Removed LNI 192.168.202.123@tcp49 [16444.092503] LNet: Removed LNI 192.168.202.123@tcp50 [16445.160797] Key type .llcrypt unregistered [16445.163246] Key type ._llcrypt unregistered [16475.060992] Key type ._llcrypt registered [16475.063160] Key type .llcrypt registered [16475.140737] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [16491.107503] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [16493.018281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16493.049657] alg: No test for adler32 (adler32-zlib) [16494.299325] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [16494.647821] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [16496.295171] Key type lgssc registered [16497.495908] Lustre: Echo OBD driver; http://www.lustre.org/ [16534.418787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16534.447315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16535.834903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16535.924864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16536.017666] Lustre: lustre-MDT0000: new disk, initializing [16536.117578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16536.146871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16541.111605] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16553.220411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16553.415777] Lustre: 506069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16553.530944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16553.542715] Lustre: Skipped 1 previous similar message [16553.673793] Lustre: lustre-MDT0001: new disk, initializing [16553.794619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16553.840169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16553.859455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16558.963559] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16569.974354] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16576.387239] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16584.916293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16585.104228] Lustre: *** cfs_fail_loc=90f, val=0*** [16585.203951] Lustre: lustre-OST0000: new disk, initializing [16585.217594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16585.228983] Lustre: 508088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16585.356184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16592.482428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16592.497146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16592.652586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16593.330569] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16603.893924] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16618.467780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16618.469686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16618.481959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16618.484799] Lustre: Skipped 1 previous similar message [16618.505683] Lustre: Skipped 1 previous similar message [16619.732526] Lustre: server umount lustre-OST0000 complete [16625.637303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16625.648794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16625.672549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16628.712068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16628.712885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16630.342862] Lustre: server umount lustre-MDT0000 complete [16634.458668] LustreError: 506061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572614 with bad export cookie 17689018800560788477 [16634.461136] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16634.474646] LustreError: 506061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16634.835475] Lustre: server umount lustre-MDT0001 complete [16646.802757] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [16650.313990] Key type lgssc unregistered [16650.727700] LNet: 509997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16650.745774] LNetError: 509997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16650.764624] LNet: Removed LNI 192.168.202.123@tcp [16652.138247] Key type .llcrypt unregistered [16652.144315] Key type ._llcrypt unregistered [16676.888398] Key type ._llcrypt registered [16676.894288] Key type .llcrypt registered [16676.988498] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [16678.480662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16678.570106] alg: No test for adler32 (adler32-zlib) [16679.740543] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [16680.075248] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [16681.767203] Key type lgssc registered [16683.346491] Lustre: Echo OBD driver; http://www.lustre.org/ [16693.381498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16693.418071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16695.033117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16699.502883] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16705.511883] LustreError: 511269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16705.537287] LustreError: 511269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16708.485058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16708.942485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16713.482145] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16722.324797] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16729.451284] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16736.496642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16736.726324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16743.364756] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16753.037929] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16768.365544] loop0: detected capacity change from 0 to 204800000 [16768.582317] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16768.616946] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16768.629177] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16776.348214] loop0: detected capacity change from 0 to 204800000 [16776.701218] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16776.731507] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16776.754730] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16782.887488] loop0: detected capacity change from 0 to 204800000 [16790.056835] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16790.247466] Lustre: fs15246-OST0000: new disk, initializing [16790.254486] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16790.262406] Lustre: 514866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16790.323432] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16797.601274] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16808.621072] loop1: detected capacity change from 0 to 204800000 [16814.266144] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16814.378168] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16814.417571] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16814.482739] Lustre: fs15246-MDT0000: new disk, initializing [16814.674724] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16814.699583] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16814.905276] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16815.968708] Lustre: 515990:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a99839a9 x1874405789584896/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787572812 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16821.136579] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [16825.137878] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16825.156958] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16825.205197] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16846.836553] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16846.846821] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16851.569032] Lustre: server umount fs15246-MDT0000 complete [16851.940825] LustreError: 512490:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16857.071034] LustreError: 511264:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16860.361582] Lustre: server umount fs15246-OST0000 complete [16872.420385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16872.423849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16872.440358] Lustre: Skipped 1 previous similar message [16872.457644] Lustre: Skipped 1 previous similar message [16877.538852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16877.552598] Lustre: Skipped 1 previous similar message [16878.333920] Lustre: server umount lustre-OST0000 complete [16882.659502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16882.665667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16882.683229] Lustre: Skipped 1 previous similar message [16882.690670] Lustre: Skipped 1 previous similar message [16887.778765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16887.790219] Lustre: Skipped 1 previous similar message [16888.872580] Lustre: server umount lustre-MDT0000 complete [16892.897310] LustreError: 511264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16892.936130] LustreError: 511264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16893.471406] LustreError: 511250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572873 with bad export cookie 2365551035595965269 [16893.478900] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16893.485185] LustreError: 511250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16893.942890] Lustre: server umount lustre-MDT0001 complete [16907.690925] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [16911.564175] Key type lgssc unregistered [16911.970141] LNet: 519103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16911.979784] LNetError: 519103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16912.022788] LNet: Removed LNI 192.168.202.123@tcp [16912.945222] Key type .llcrypt unregistered [16912.949362] Key type ._llcrypt unregistered [16935.982359] Key type ._llcrypt registered [16935.983934] Key type .llcrypt registered [16936.119941] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [16937.081057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16937.110319] alg: No test for adler32 (adler32-zlib) [16938.567239] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [16938.964646] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [16940.631171] Key type lgssc registered [16942.060762] Lustre: Echo OBD driver; http://www.lustre.org/ [16984.366218] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [16995.323736] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [17033.769845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17033.794275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17035.033337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17035.081677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17035.198785] Lustre: lustre-MDT0000: new disk, initializing [17035.336744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17035.375822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17039.949287] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [17051.838949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17051.984211] Lustre: 523574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17052.032697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17052.036126] Lustre: Skipped 1 previous similar message [17052.138900] Lustre: lustre-MDT0001: new disk, initializing [17052.282579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17052.350373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17052.368804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17058.594990] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [17069.517310] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17076.322196] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17084.292660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17084.545549] Lustre: lustre-OST0000: new disk, initializing [17084.550806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17084.557290] Lustre: 525543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17084.669052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17091.481540] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [17092.693451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17092.715288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17092.828923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17101.526888] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17106.542409] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17106.778888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17111.308729] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17111.521252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17113.576326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17113.577692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17113.590595] LustreError: Skipped 1 previous similar message [17113.610725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17118.712703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17118.724457] Lustre: Skipped 2 previous similar messages [17119.625473] Lustre: server umount lustre-OST0000 complete [17128.928378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17128.943419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17128.946510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17128.957023] Lustre: Skipped 2 previous similar messages [17130.701224] Lustre: server umount lustre-MDT0000 complete [17134.610964] LustreError: 524377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573114 with bad export cookie 14919813435771912420 [17134.616390] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17134.631474] LustreError: 524377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17134.982382] Lustre: server umount lustre-MDT0001 complete [17168.367459] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [17180.431425] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [17216.725690] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16928 sec ========= 08:06:35 (1787573195) [17219.454365] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:06:37 (1787573197) === [17223.879332] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:06:42 (1787573202) === [17270.641873] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [17273.892342] Key type lgssc unregistered [17274.175696] LNet: 532196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17274.185670] LNetError: 532196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17274.222148] LNet: Removed LNI 192.168.202.123@tcp [17275.174455] Key type .llcrypt unregistered [17275.176660] Key type ._llcrypt unregistered