[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 573066135 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006776] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009144] LSM: Security Framework initializing [ 0.010040] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.019719] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023513] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024293] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025169] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026196] *** VALIDATE tmpfs *** [ 0.027568] *** VALIDATE proc *** [ 0.029409] *** VALIDATE cgroup *** [ 0.030014] *** VALIDATE cgroup2 *** [ 0.032225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039120] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049007] ... event mask: 000000070000000f [ 0.050350] rcu: Hierarchical SRCU implementation. [ 0.053289] smp: Bringing up secondary CPUs ... [ 0.055105] x86: Booting SMP configuration: [ 0.056011] .... node #0, CPUs: #1 #2 #3 [ 0.068193] smp: Brought up 1 node, 4 CPUs [ 0.070016] smpboot: Max logical packages: 1 [ 0.071015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101155] node 0 deferred pages initialised in 28ms [ 0.105201] devtmpfs: initialized [ 0.106286] x86/mm: Memory block size: 128MB [ 0.109116] gcov: version magic: 0x41383552 [ 0.111346] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113305] pinctrl core: initialized pinctrl subsystem [ 0.114186] [ 0.114642] ************************************************************* [ 0.115019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.116014] ** ** [ 0.117014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.118011] ** ** [ 0.119012] ** This means that this kernel is built to expose internal ** [ 0.120012] ** IOMMU data structures, which may compromise security on ** [ 0.121012] ** your system. ** [ 0.122010] ** ** [ 0.123010] ** If you see this message and you are not debugging the ** [ 0.124012] ** kernel, report this immediately to your vendor! ** [ 0.125010] ** ** [ 0.126010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127013] ************************************************************* [ 0.128878] NET: Registered protocol family 16 [ 0.129643] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.130061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.131077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.132766] cpuidle: using governor menu [ 0.133996] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137637] PCI: Using configuration type 1 for base access [ 0.139139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.150146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.152015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.156289] cryptd: max_cpu_qlen set to 1000 [ 0.158272] ACPI: Added _OSI(Module Device) [ 0.159014] ACPI: Added _OSI(Processor Device) [ 0.160013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.161010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.165321] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.168734] ACPI: Interpreter enabled [ 0.170108] ACPI: PM: (supports S0 S3 S4 S5) [ 0.172013] ACPI: Using IOAPIC for interrupt routing [ 0.175166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.179397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.192799] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.195147] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.199022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.208627] acpiphp: Slot [2] registered [ 0.211143] acpiphp: Slot [5] registered [ 0.212095] acpiphp: Slot [6] registered [ 0.214088] acpiphp: Slot [7] registered [ 0.216326] acpiphp: Slot [8] registered [ 0.218151] acpiphp: Slot [9] registered [ 0.220126] acpiphp: Slot [10] registered [ 0.222122] acpiphp: Slot [3] registered [ 0.224119] acpiphp: Slot [4] registered [ 0.226097] acpiphp: Slot [11] registered [ 0.228087] acpiphp: Slot [12] registered [ 0.230088] acpiphp: Slot [13] registered [ 0.231199] acpiphp: Slot [14] registered [ 0.233116] acpiphp: Slot [15] registered [ 0.235144] acpiphp: Slot [16] registered [ 0.237142] acpiphp: Slot [17] registered [ 0.239134] acpiphp: Slot [18] registered [ 0.241115] acpiphp: Slot [19] registered [ 0.243088] acpiphp: Slot [20] registered [ 0.244074] acpiphp: Slot [21] registered [ 0.245080] acpiphp: Slot [22] registered [ 0.247089] acpiphp: Slot [23] registered [ 0.249143] acpiphp: Slot [24] registered [ 0.251091] acpiphp: Slot [25] registered [ 0.252208] acpiphp: Slot [26] registered [ 0.254198] acpiphp: Slot [27] registered [ 0.256089] acpiphp: Slot [28] registered [ 0.258091] acpiphp: Slot [29] registered [ 0.260073] acpiphp: Slot [30] registered [ 0.262088] acpiphp: Slot [31] registered [ 0.264097] PCI host bridge to bus 0000:00 [ 0.265016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.283187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290463] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.300920] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.305000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306063] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.312017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315542] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318945] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325896] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356205] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.369018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.388016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.396649] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.405015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.430015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439642] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.446017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.452013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.480475] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.487016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.493016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.508014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.514445] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.520017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.528016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.539015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.547569] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.553060] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.557016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.573016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.585759] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.588331] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.590334] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.593382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.595227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600070] iommu: Default domain type: Passthrough [ 0.601000] SCSI subsystem initialized [ 0.602114] ACPI: bus type USB registered [ 0.604074] usbcore: registered new interface driver usbfs [ 0.606053] usbcore: registered new interface driver hub [ 0.608092] usbcore: registered new device driver usb [ 0.609119] pps_core: LinuxPPS API ver. 1 registered [ 0.611007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.614047] PTP clock support registered [ 0.617109] EDAC MC: Ver: 3.0.0 [ 0.618109] PCI: Using ACPI for IRQ routing [ 0.620069] NetLabel: Initializing [ 0.622007] NetLabel: domain hash size = 128 [ 0.623005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626075] NetLabel: unlabeled traffic allowed by default [ 0.628392] vgaarb: loaded [ 0.629708] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.631007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.638142] clocksource: Switched to clocksource kvm-clock [ 0.781388] VFS: Disk quotas dquot_6.6.0 [ 0.784030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.787222] *** VALIDATE ramfs *** [ 0.789629] *** VALIDATE hugetlbfs *** [ 0.795321] pnp: PnP ACPI init [ 0.798747] pnp: PnP ACPI: found 6 devices [ 0.819283] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.822528] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.825013] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.827487] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.830071] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832518] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.835561] NET: Registered protocol family 2 [ 0.838192] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.842925] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.845514] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.850403] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853448] TCP: Hash tables configured (established 65536 bind 65536) [ 0.855578] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.858112] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.861060] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864648] NET: Registered protocol family 1 [ 0.867646] RPC: Registered named UNIX socket transport module. [ 0.869900] RPC: Registered udp transport module. [ 0.871686] RPC: Registered tcp transport module. [ 0.873385] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.875719] NET: Registered protocol family 44 [ 0.877227] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.879325] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.880865] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.883303] PCI: CLS 0 bytes, default 64 [ 0.885814] Unpacking initramfs... [ 2.665771] debug: unmapping init [mem 0xffff91d9fcc54000-0xffff91d9fffbffff] [ 2.669957] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.672843] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.676540] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.626128] Initialise system trusted keyrings [ 3.627868] Key type blacklist registered [ 3.629739] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.639216] zbud: loaded [ 3.641890] *** VALIDATE nfs *** [ 3.642984] *** VALIDATE nfs4 *** [ 3.644574] pstore: using deflate compression [ 3.647912] Platform Keyring initialized [ 3.867385] NET: Registered protocol family 38 [ 3.870644] Key type asymmetric registered [ 3.873403] Asymmetric key parser 'x509' registered [ 3.877730] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.882472] io scheduler mq-deadline registered [ 3.884212] io scheduler kyber registered [ 3.886913] io scheduler bfq registered [ 3.889456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.892707] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.898165] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.905594] ACPI: Power Button [PWRF] [ 3.912819] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.920143] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.936727] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.942709] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.955179] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.990916] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.027917] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.032181] Non-volatile memory driver v1.3 [ 4.033716] Linux agpgart interface v0.103 [ 4.068970] virtio_blk virtio1: [vda] 145848 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.072359] vda: detected capacity change from 0 to 74674176 [ 4.085964] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.088722] vdb: detected capacity change from 0 to 1073741824 [ 4.102200] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.105040] vdc: detected capacity change from 0 to 2621440000 [ 4.118624] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.121327] vdd: detected capacity change from 0 to 2621440000 [ 4.134501] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.137458] vde: detected capacity change from 0 to 4294967296 [ 4.150559] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.153327] vdf: detected capacity change from 0 to 4294967296 [ 4.159104] libphy: Fixed MDIO Bus: probed [ 4.163641] usbcore: registered new interface driver usbserial_generic [ 4.165967] usbserial: USB Serial support registered for generic [ 4.167966] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.171539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.173089] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.175388] mousedev: PS/2 mouse device common for all mice [ 4.178203] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.181210] rtc_cmos 00:05: RTC can wake from S4 [ 4.185355] rtc_cmos 00:05: registered as rtc0 [ 4.187809] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.190890] intel_pstate: CPU model not supported [ 4.191058] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.197519] hid: raw HID events driver (C) Jiri Kosina [ 4.199680] usbcore: registered new interface driver usbhid [ 4.202282] usbhid: USB HID core driver [ 4.203503] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.203705] drop_monitor: Initializing network drop monitor service [ 4.208426] Initializing XFRM netlink socket [ 4.210377] NET: Registered protocol family 10 [ 4.212886] Segment Routing with IPv6 [ 4.213826] NET: Registered protocol family 17 [ 4.215498] mpls_gso: MPLS GSO support [ 4.222373] RAS: Correctable Errors collector initialized. [ 4.225105] AVX version of gcm_enc/dec engaged. [ 4.226742] AES CTR mode by8 optimization enabled [ 4.307977] sched_clock: Marking stable (4307900687, 0)->(5715003185, -1407102498) [ 4.311789] registered taskstats version 1 [ 4.313619] Loading compiled-in X.509 certificates [ 4.315494] zswap: loaded using pool lzo/zbud [ 4.338913] Key type big_key registered [ 4.350737] Key type encrypted registered [ 4.352255] ima: No TPM chip found, activating TPM-bypass! [ 4.354566] ima: Allocated hash algorithm: sha1 [ 4.356141] ima: No architecture policies found [ 4.357869] evm: Initialising EVM extended attributes: [ 4.362802] evm: security.selinux [ 4.363868] evm: security.ima [ 4.364943] evm: security.capability [ 4.366297] evm: HMAC attrs: 0x1 [ 4.368764] rtc_cmos 00:05: setting system clock to 2026-09-02 12:44:54 UTC (1788353094) [ 4.375047] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 4.378098] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 4.385129] Write protecting the kernel read-only data: 28672k [ 4.389207] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 4.391824] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 4.447790] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.459154] systemd[1]: Detected virtualization kvm. [ 4.463588] systemd[1]: Detected architecture x86-64. [ 4.466432] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.492105] systemd[1]: No hostname configured. [ 4.494122] systemd[1]: Set hostname to . [ 4.496375] random: systemd: uninitialized urandom read (16 bytes read) [ 4.499222] systemd[1]: Initializing machine ID from random generator. [ 4.577586] random: ln: uninitialized urandom read (6 bytes read) [ 4.660921] random: systemd: uninitialized urandom read (16 bytes read) [ 4.663721] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.676132] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.685760] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.651722] device-mapper: uevent: version 1.0.3 [ 5.653969] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 6.847214] random: fast init done [ 6.863724] virtio_net virtio0 ens2: renamed from eth0 [ 7.155368] scsi host0: ata_piix [ 7.197130] scsi host1: ata_piix [ 7.198507] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.202867] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.884737] random: crng init done [ 11.886277] random: 7 urandom warning(s) missed due to ratelimiting [ 12.260533] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.956227] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.638501] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.216329] SELinux: Disabled at runtime. [ 16.292855] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.309504] systemd[1]: Detected virtualization kvm. [ 16.311632] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.727212] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.730860] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.746259] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.753950] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.758473] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.765863] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.777856] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 17.965250] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.861206] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.561505] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.588960] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.897141] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.995436] EDAC sbridge: Ver: 1.1.2 [ 22.586788] Key type dns_resolver registered [ 22.945234] NFS: Registering the id_resolver key type [ 22.947364] Key type id_resolver registered [ 22.949102] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 47.558833] libcfs: loading out-of-tree module taints kernel. [ 47.577137] Key type ._llcrypt registered [ 47.578664] Key type .llcrypt registered [ 47.627924] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 55.036934] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 55.700485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.709386] alg: No test for adler32 (adler32-zlib) [ 56.684193] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 56.978631] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 58.583204] Key type lgssc registered [ 59.157288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.364425] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 79.680391] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 84.053805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.061779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.144876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.157412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.199265] Lustre: lustre-MDT0000: new disk, initializing [ 85.228376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.236213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.507662] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.605225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 91.640785] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 91.652889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 91.656078] Lustre: Skipped 1 previous similar message [ 91.691295] Lustre: lustre-MDT0001: new disk, initializing [ 91.710132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 91.717434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 91.722283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 92.955911] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.198447] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.259878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.352945] Lustre: lustre-OST0000: new disk, initializing [ 98.355400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.358800] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.377322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.172187] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.403432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.407451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.431447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 105.523756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 105.568131] Lustre: lustre-OST0001: new disk, initializing [ 105.569896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 105.572097] Lustre: 9487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 105.588981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.374160] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.568499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 110.571778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 110.584181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 113.020754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 117.647726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 119.605784] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 121.161095] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 122.819521] Lustre: DEBUG MARKER: Client: 2.17.55.23 [ 123.825761] Lustre: DEBUG MARKER: MDS: 2.17.55.23 [ 124.755259] Lustre: DEBUG MARKER: OSS: 2.17.55.23 [ 125.345358] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 08:46:55 EDT 2026 [ 131.105950] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 131.611136] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 146.399895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.400223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 146.405983] Lustre: Skipped 3 previous similar messages [ 146.407715] Lustre: Skipped 2 previous similar messages [ 149.462625] Lustre: server umount lustre-MDT0000 complete [ 150.669718] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353240 with bad export cookie 14341676264043767725 [ 150.671332] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 150.673391] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 151.519715] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.519930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 151.523147] Lustre: Skipped 1 previous similar message [ 151.527890] Lustre: Skipped 2 previous similar messages [ 155.551493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 155.553429] Lustre: Skipped 1 previous similar message [ 156.832990] Lustre: server umount lustre-MDT0001 complete [ 164.219251] Lustre: server umount lustre-OST0000 complete [ 171.531903] Lustre: server umount lustre-OST0001 complete [ 173.791106] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 177.491147] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 190.229868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.317758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.331759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.372074] Lustre: lustre-MDT0000: new disk, initializing [ 190.405875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.415559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.905500] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 196.284858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.330857] Lustre: 15025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.447863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.865479] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 201.758345] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 203.838118] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 206.290518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.397166] Lustre: lustre-OST0000: new disk, initializing [ 206.399405] Lustre: Skipped 1 previous similar message [ 206.401381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.404353] Lustre: Skipped 2 previous similar messages [ 206.407261] Lustre: 16991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.446354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.449171] Lustre: Skipped 1 previous similar message [ 208.126574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.130315] Lustre: Skipped 1 previous similar message [ 208.132954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.167629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.495274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 212.250473] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 213.713783] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 213.788374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 215.149589] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 215.229272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 218.592742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 218.595024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.595597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 218.605111] Lustre: Skipped 1 previous similar message [ 223.712843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 223.715802] Lustre: Skipped 2 previous similar messages [ 230.368301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 230.433437] Lustre: server umount lustre-OST0000 complete [ 232.415928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 232.418745] LustreError: Skipped 1 previous similar message [ 232.420422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.425918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 232.430157] Lustre: Skipped 2 previous similar messages [ 237.850861] Lustre: server umount lustre-MDT0000 complete [ 239.072985] LustreError: 15035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 239.078550] LustreError: 15035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 239.104550] LustreError: 15820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353329 with bad export cookie 14341676264043769244 [ 239.105963] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 239.109903] LustreError: 15820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 239.226252] Lustre: server umount lustre-MDT0001 complete [ 240.647950] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:48:50 (1788353330) [ 241.561164] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 242.254418] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:48:52 (1788353332) [ 244.683583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.869885] LustreError: 19400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 244.893183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.367289] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 249.808112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.546522] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 254.289970] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 256.416146] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 258.913200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.038226] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 263.789633] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 280.547573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.549534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 280.553095] Lustre: Skipped 1 previous similar message [ 280.557810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 280.562421] Lustre: Skipped 3 previous similar messages [ 283.559799] Lustre: server umount lustre-OST0000 complete [ 291.296846] Lustre: server umount lustre-MDT0000 complete [ 292.791930] LustreError: 19380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353382 with bad export cookie 14341676264043770126 [ 292.793936] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.801075] LustreError: 19380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 292.918561] Lustre: server umount lustre-MDT0001 complete [ 297.532229] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 298.746317] Key type lgssc unregistered [ 298.904771] LNet: 22951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.910095] LNetError: 22951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.918394] LNet: Removed LNI 192.168.203.142@tcp [ 299.335191] Key type .llcrypt unregistered [ 299.336749] Key type ._llcrypt unregistered [ 309.605186] Key type ._llcrypt registered [ 309.606981] Key type .llcrypt registered [ 309.651823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 310.090617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 310.107950] alg: No test for adler32 (adler32-zlib) [ 310.970891] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 311.067741] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 312.655135] Key type lgssc registered [ 313.045088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 316.020117] loop: module loaded [ 316.039288] loop0: detected capacity change from 0 to 204800000 [ 316.170701] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.177605] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.309534] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.436339] loop0: detected capacity change from 0 to 204800000 [ 318.496530] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.501088] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.506984] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 320.494064] loop0: detected capacity change from 0 to 204800000 [ 322.583876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 322.592041] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.682437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 323.697497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 323.736753] Lustre: lustre-MDT0000: new disk, initializing [ 323.771351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.778900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 325.279815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 329.174914] loop1: detected capacity change from 0 to 204800000 [ 331.192232] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 331.304678] Lustre: lustre-OST07c6: new disk, initializing [ 331.306798] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 331.309594] Lustre: Skipped 1 previous similar message [ 331.312435] Lustre: 26111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 331.345756] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 333.407401] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 335.875692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 335.879372] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 335.890378] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 341.012900] Lustre: 25099:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 341.017639] Lustre: 25099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 341.020546] Lustre: 25099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 341.023743] Lustre: 25099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 341.026747] Lustre: 25099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 341.029711] Lustre: 25099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 356.087489] Lustre: 25099:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 356.092194] Lustre: 25099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 356.094014] Lustre: 25099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 356.096364] Lustre: 25099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 356.099019] Lustre: 25099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 356.101162] Lustre: 25099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 359.574165] Lustre: 26629:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 359.577248] Lustre: 26629:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 359.579323] Lustre: 26629:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 359.581498] Lustre: 26629:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 359.583557] Lustre: 26629:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 359.585924] Lustre: 26629:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 359.588300] Lustre: 26629:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 359.591022] Lustre: 26629:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 359.593234] Lustre: 26629:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 359.595664] Lustre: 26629:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 359.598012] Lustre: 26629:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 359.600130] Lustre: 26629:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 362.650804] Lustre: 26632:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 362.655220] Lustre: 26632:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 362.658883] Lustre: 26632:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 362.662207] Lustre: 26632:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 362.664605] Lustre: 26632:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 362.667175] Lustre: 26632:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 362.669386] Lustre: 26632:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 362.671499] Lustre: 26632:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 362.673605] Lustre: 26632:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 362.675590] Lustre: 26632:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 362.677791] Lustre: 26632:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 362.679856] Lustre: 26632:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 369.160334] Lustre: 25100:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 369.163119] Lustre: 25100:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 369.165432] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 369.167619] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 369.169751] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 369.172014] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 369.174020] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 369.176232] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 369.178410] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 369.180565] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 369.182760] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 369.184813] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 378.692668] Lustre: 25100:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 378.696984] Lustre: 25100:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 378.699515] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 378.702129] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 378.704122] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 378.706190] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 378.708476] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 378.711430] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 378.714754] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 378.716535] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 378.719034] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 378.722092] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.879989] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 422.883438] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.888378] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 425.352333] Lustre: server umount lustre-OST07c6 complete [ 428.323877] Lustre: server umount lustre-MDT0000 complete [ 433.689994] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 434.720641] Key type lgssc unregistered [ 434.861475] LNet: 28694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.864269] LNetError: 28694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.872366] LNet: Removed LNI 192.168.203.142@tcp [ 435.213885] Key type .llcrypt unregistered [ 435.215128] Key type ._llcrypt unregistered [ 442.127418] Key type ._llcrypt registered [ 442.128703] Key type .llcrypt registered [ 442.161641] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 442.464379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 442.501056] alg: No test for adler32 (adler32-zlib) [ 443.359850] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 443.436851] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 445.007165] Key type lgssc registered [ 445.355174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 447.875160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 447.880753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.008046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.232227] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 452.927855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.047325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.347276] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 456.654294] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.349342] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.426619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.529950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.319800] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 464.552856] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 474.852300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 480.228039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.229437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.233510] Lustre: Skipped 1 previous similar message [ 480.238538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.347445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.349835] Lustre: Skipped 1 previous similar message [ 485.709728] Lustre: server umount lustre-OST0000 complete [ 488.927990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 488.930804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.935806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 488.938365] Lustre: Skipped 1 previous similar message [ 490.466065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.215735] Lustre: server umount lustre-MDT0000 complete [ 494.418218] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353584 with bad export cookie 17418635695568630302 [ 494.421702] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.422822] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 494.530161] Lustre: server umount lustre-MDT0001 complete [ 498.022117] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 499.004185] Key type lgssc unregistered [ 499.132371] LNet: 33690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.135542] LNetError: 33690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 499.143291] LNet: Removed LNI 192.168.203.142@tcp [ 499.432411] Key type .llcrypt unregistered [ 499.433577] Key type ._llcrypt unregistered [ 506.707763] Key type ._llcrypt registered [ 506.709362] Key type .llcrypt registered [ 506.748392] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 507.179107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 507.187123] alg: No test for adler32 (adler32-zlib) [ 508.045281] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 508.138159] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 509.719148] Key type lgssc registered [ 510.093123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 512.786944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 512.792853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.947064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.327966] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 518.324996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.458199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 519.859616] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 522.315157] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.136329] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 526.513632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 526.635101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.578928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 531.001157] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 538.917743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 549.344596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.345319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.347137] LustreError: Skipped 1 previous similar message [ 549.353381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.355632] Lustre: Skipped 1 previous similar message [ 551.448296] Lustre: server umount lustre-OST0000 complete [ 554.464572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 554.465122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.472762] Lustre: Skipped 1 previous similar message [ 554.474979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.476771] Lustre: Skipped 1 previous similar message [ 558.805499] Lustre: server umount lustre-MDT0000 complete [ 559.584940] LustreError: 35677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.592426] LustreError: 35677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 559.962685] LustreError: 34917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353650 with bad export cookie 13415732153947498751 [ 559.966033] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.966507] LustreError: 34917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 560.076367] Lustre: server umount lustre-MDT0001 complete [ 563.639039] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 564.688623] Key type lgssc unregistered [ 564.826429] LNet: 38490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.829032] LNetError: 38490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.835290] LNet: Removed LNI 192.168.203.142@tcp [ 565.130398] Key type .llcrypt unregistered [ 565.131949] Key type ._llcrypt unregistered [ 573.789840] Key type ._llcrypt registered [ 573.791446] Key type .llcrypt registered [ 573.829767] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 574.216302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 574.252123] alg: No test for adler32 (adler32-zlib) [ 575.114506] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 575.218723] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 576.799182] Key type lgssc registered [ 577.195057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 579.663784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 579.670924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.812358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.160586] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 584.755323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.875620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 586.214245] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 588.716434] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 590.577847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 592.844908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.951887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.810256] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 597.048199] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.221582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 610.274734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.276187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.279304] Lustre: Skipped 1 previous similar message [ 611.996977] Lustre: server umount lustre-MDT0000 complete [ 612.829369] LustreError: 42369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.835184] LustreError: 42369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 613.166265] LustreError: 39865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353703 with bad export cookie 16413826590954823392 [ 613.169159] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.171558] LustreError: 39865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.392053] LustreError: 39877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.392391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.399073] Lustre: Skipped 1 previous similar message [ 615.400444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 615.402901] Lustre: Skipped 2 previous similar messages [ 617.940687] LustreError: 41313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.948762] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 619.363888] Lustre: server umount lustre-MDT0001 complete [ 637.132536] Lustre: server umount lustre-OST0000 complete [ 642.385884] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 643.383820] Key type lgssc unregistered [ 643.503431] LNet: 43758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.506336] LNetError: 43758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.516356] LNet: Removed LNI 192.168.203.142@tcp [ 643.823656] Key type .llcrypt unregistered [ 643.824645] Key type ._llcrypt unregistered [ 652.711819] Key type ._llcrypt registered [ 652.712971] Key type .llcrypt registered [ 652.751672] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 653.086111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 653.116875] alg: No test for adler32 (adler32-zlib) [ 653.968919] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 654.058177] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 655.639103] Key type lgssc registered [ 656.001584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 658.552848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 658.561724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.717978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.051246] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 663.822690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.941203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 665.271369] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 667.619670] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 669.435424] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 671.689410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.803297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.709884] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 676.041452] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 684.134374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 689.635443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.636559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.636989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 689.640248] Lustre: Skipped 1 previous similar message [ 689.643610] LustreError: Skipped 1 previous similar message [ 691.674274] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 691.677966] Lustre: Skipped 1 previous similar message [ 694.754177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.757216] Lustre: Skipped 1 previous similar message [ 694.863992] Lustre: server umount lustre-OST0000 complete [ 707.365222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.366316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 707.374467] Lustre: Skipped 1 previous similar message [ 712.672133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.674832] Lustre: Skipped 2 previous similar messages [ 713.622558] Lustre: server umount lustre-MDT0000 complete [ 714.818868] LustreError: 45132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353804 with bad export cookie 3237685022396405401 [ 714.822067] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.822540] LustreError: 45132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 720.997313] Lustre: server umount lustre-MDT0001 complete [ 724.350531] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 725.295198] Key type lgssc unregistered [ 725.423364] LNet: 48875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.425495] LNetError: 48875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.434245] LNet: Removed LNI 192.168.203.142@tcp [ 725.719185] Key type .llcrypt unregistered [ 725.720436] Key type ._llcrypt unregistered [ 733.928904] Key type ._llcrypt registered [ 733.930092] Key type .llcrypt registered [ 733.963716] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 734.286826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.316076] alg: No test for adler32 (adler32-zlib) [ 735.163928] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 735.243319] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 736.823173] Key type lgssc registered [ 737.171857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.578834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 739.585017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.716097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.029381] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 744.759391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.877596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 746.201848] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 748.383101] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 750.108072] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 752.192654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.301258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.023749] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 756.272949] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.604866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 768.956643] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 770.040017] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 780.775104] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-02 8h57m50s [ 781.253072] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 782.299915] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 783.316153] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 785.887740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 785.889992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.893834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 788.049901] Lustre: server umount lustre-MDT0000 complete [ 789.100324] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 789.459963] LustreError: 50264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 789.467817] LustreError: 50264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 791.147491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.187186] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.269719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.299608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 792.292847] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 792.496374] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 793.518089] Lustre: DEBUG MARKER: Wait for df (36557) ... [ 793.984069] Lustre: DEBUG MARKER: done [ 795.615853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 795.618280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.622538] Lustre: Skipped 2 previous similar messages [ 795.624808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.627432] Lustre: Skipped 5 previous similar messages [ 796.639992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.642713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.974287] Lustre: server umount lustre-OST0000 complete [ 802.784513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.784563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 802.789616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.793600] Lustre: Skipped 4 previous similar messages [ 808.274377] Lustre: server umount lustre-MDT0000 complete [ 808.864215] LustreError: 50269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.871121] LustreError: 50269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 809.453205] LustreError: 50248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788353899 with bad export cookie 13558081714608679905 [ 809.456424] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.458101] LustreError: 50248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 809.565520] Lustre: server umount lustre-MDT0001 complete [ 812.839875] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 813.795644] Key type lgssc unregistered [ 813.915897] LNet: 55200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.918480] LNetError: 55200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.930341] LNet: Removed LNI 192.168.203.142@tcp [ 814.208105] Key type .llcrypt unregistered [ 814.209629] Key type ._llcrypt unregistered [ 827.554179] Key type ._llcrypt registered [ 827.555920] Key type .llcrypt registered [ 827.593698] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 827.925673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.974834] alg: No test for adler32 (adler32-zlib) [ 828.818029] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 828.897892] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 830.471199] Key type lgssc registered [ 830.851939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 833.324402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 833.331451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.407877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 834.413984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 834.468154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.693199] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 838.388775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.417933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 838.428284] Lustre: 57385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 838.513108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.846685] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 842.211841] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 843.978970] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 846.151272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.222225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 846.279985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.089068] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 849.489072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 850.418920] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 853.667676] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-02 8h59m03s [ 854.171817] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 856.380486] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 857.162330] Lustre: DEBUG MARKER: Creating a test file and stat it [ 857.183658] LustreError: 58647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 877.535907] Lustre: lustre-MDT0000: Client 9ca4b002-274d-400b-abf6-af4ab3bbc475 (at 192.168.203.42@tcp) reconnecting [ 897.231125] LustreError: 58647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 897.986412] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 898.864317] Lustre: DEBUG MARKER: done [ 900.576360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.576910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.579093] LustreError: Skipped 1 previous similar message [ 900.579622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.582901] Lustre: Skipped 1 previous similar message [ 905.697201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.699518] Lustre: Skipped 2 previous similar messages [ 906.635077] Lustre: server umount lustre-OST0000 complete [ 910.303880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 910.307387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.312320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.004527] Lustre: server umount lustre-MDT0000 complete [ 915.190595] LustreError: 57375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354005 with bad export cookie 12476060835663137905 [ 915.195664] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.199811] LustreError: 57375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 915.324453] Lustre: server umount lustre-MDT0001 complete [ 918.754934] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 919.807244] Key type lgssc unregistered [ 919.948352] LNet: 61711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.951283] LNetError: 61711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.959291] LNet: Removed LNI 192.168.203.142@tcp [ 920.279098] Key type .llcrypt unregistered [ 920.280264] Key type ._llcrypt unregistered [ 935.512387] Key type ._llcrypt registered [ 935.513491] Key type .llcrypt registered [ 935.550404] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 935.827236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.844906] alg: No test for adler32 (adler32-zlib) [ 936.698587] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 936.784840] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 938.359131] Key type lgssc registered [ 938.695477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.710673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.717285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.794211] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 940.801287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 940.856346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.190840] Lustre: Failing over lustre-MDT0000 [ 941.294629] Lustre: server umount lustre-MDT0000 complete [ 943.296797] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:00:33 (1788354033) [ 945.460152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.594433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.844973] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 949.522489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.559086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 949.570541] Lustre: 64485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 949.667206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 950.968588] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 953.295571] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 955.175934] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 957.339070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.403307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 957.473993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.350071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 959.250479] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 961.526848] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.483606] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 968.673792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.674741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.677595] Lustre: Skipped 2 previous similar messages [ 968.679492] Lustre: Skipped 1 previous similar message [ 972.248695] Lustre: server umount lustre-MDT0000 complete [ 973.443058] LustreError: 64516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354063 with bad export cookie 16531792422850828419 [ 973.445980] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.447636] LustreError: 64516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 973.551296] Lustre: server umount lustre-MDT0001 complete [ 974.648758] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 976.117942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 978.830572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.007721] LustreError: 63408:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8b07fbd x1875225111198464/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 983.135624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.163721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 984.344657] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 986.970872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.064853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.140436] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 988.494240] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 989.152095] Lustre: 63412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788354063/real 1788354063] req@000000001414f517 x1875225111198208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788354079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 990.794723] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.481954] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 995.167188] Lustre: 63410:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788354069/real 1788354069] req@000000000d2155d6 x1875225111198336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788354085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 997.344303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 997.347747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.352198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.369403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.371198] Lustre: Skipped 2 previous similar messages [ 999.391119] Lustre: 63409:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788354073/real 1788354073] req@000000004dc81c25 x1875225111198592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788354089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1000.275620] Lustre: server umount lustre-MDT0000 complete [ 1001.474138] LustreError: 65219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354091 with bad export cookie 16531792422850830687 [ 1001.476648] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.479572] LustreError: 65219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1001.579281] Lustre: server umount lustre-MDT0001 complete [ 1003.563426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1006.500918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.615743] LustreError: 63408:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003c0c7e15 x1875225111218432/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1011.748483] LustreError: 68992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.754245] LustreError: 68992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1011.771415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.773438] Lustre: Skipped 1 previous similar message [ 1011.792188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1013.155397] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1015.914027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.991672] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.995742] Lustre: Skipped 2 previous similar messages [ 1017.288850] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1018.084878] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1018.087517] Lustre: Skipped 1 previous similar message [ 1019.103187] Lustre: 63412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788354093/real 1788354093] req@00000000aca9dfa9 x1875225111218048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788354109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1019.463793] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1021.151170] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1023.457447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.458369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.461579] Lustre: Skipped 1 previous similar message [ 1024.479153] Lustre: 63410:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788354098/real 1788354098] req@00000000a331ef95 x1875225111218304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788354114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1024.486179] Lustre: 63410:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1028.576623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.578647] Lustre: Skipped 4 previous similar messages [ 1028.823711] Lustre: server umount lustre-MDT0000 complete [ 1030.066022] LustreError: 65219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354120 with bad export cookie 16531792422850832066 [ 1030.069453] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.069799] LustreError: 65219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1030.174876] Lustre: server umount lustre-MDT0001 complete [ 1032.268315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1033.358517] Lustre: DEBUG MARKER: files compared the same [ 1044.322065] Lustre: server umount lustre-OST0000 complete [ 1049.706940] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1050.656278] Key type lgssc unregistered [ 1050.771354] LNet: 74210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.775190] LNetError: 74210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.783401] LNet: Removed LNI 192.168.203.142@tcp [ 1051.066401] Key type .llcrypt unregistered [ 1051.067676] Key type ._llcrypt unregistered [ 1059.509804] Key type ._llcrypt registered [ 1059.511485] Key type .llcrypt registered [ 1059.550192] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1059.970628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.978851] alg: No test for adler32 (adler32-zlib) [ 1060.829919] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1060.912344] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1062.487139] Key type lgssc registered [ 1062.824973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.255846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.261682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.387362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.588237] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug +malloc all [ 1070.239531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.337632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.612508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug +malloc all [ 1073.799145] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1075.524164] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.685205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.774325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.630980] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug +malloc all [ 1081.935714] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.105078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1096.159800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.160289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.160464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.164980] Lustre: Skipped 1 previous similar message [ 1101.000563] Lustre: server umount lustre-OST0000 complete [ 1106.399554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.399652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1106.403221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.405603] LustreError: Skipped 1 previous similar message [ 1106.408767] Lustre: Skipped 2 previous similar messages [ 1108.309565] Lustre: server umount lustre-MDT0000 complete [ 1109.517616] LustreError: 75583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354199 with bad export cookie 1070622054251994275 [ 1109.518805] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.521969] LustreError: 75583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1109.621414] Lustre: server umount lustre-MDT0001 complete [ 1113.388074] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1114.403951] Key type lgssc unregistered [ 1114.532519] LNet: 79152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.534799] LNetError: 79152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.542477] LNet: Removed LNI 192.168.203.142@tcp [ 1114.886493] Key type .llcrypt unregistered [ 1114.887932] Key type ._llcrypt unregistered [ 1123.649966] Key type ._llcrypt registered [ 1123.651402] Key type .llcrypt registered [ 1123.697951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1124.095276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.125100] alg: No test for adler32 (adler32-zlib) [ 1125.003950] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1125.113701] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1126.703181] Key type lgssc registered [ 1127.085530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.494262] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.500305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.927386] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1150.047346] LustreError: 80521:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 1176.671235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.424958] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1180.749990] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1183.141884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.174911] Lustre: *** cfs_fail_loc=706, val=0*** [ 1183.293105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.318718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1184.547646] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1187.378635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.522892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.879729] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1191.107596] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1192.726655] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1193.439837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.442586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.447071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.026281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.026772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.438023] Lustre: server umount lustre-OST0000 complete [ 1202.144605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.149522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.151262] Lustre: Skipped 2 previous similar messages [ 1206.677762] Lustre: server umount lustre-MDT0000 complete [ 1207.263931] LustreError: 81608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.268262] LustreError: 81608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1207.845372] LustreError: 80527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354297 with bad export cookie 16610195532325991876 [ 1207.847818] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.849277] LustreError: 80527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1207.968262] Lustre: server umount lustre-MDT0001 complete [ 1211.327769] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1212.330320] Key type lgssc unregistered [ 1212.458408] LNet: 84129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.461179] LNetError: 84129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.469317] LNet: Removed LNI 192.168.203.142@tcp [ 1212.761682] Key type .llcrypt unregistered [ 1212.762911] Key type ._llcrypt unregistered [ 1221.803298] Key type ._llcrypt registered [ 1221.804476] Key type .llcrypt registered [ 1221.843916] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1222.269520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.296135] alg: No test for adler32 (adler32-zlib) [ 1223.158274] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1223.248922] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1224.831144] Key type lgssc registered [ 1225.211746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1227.848822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1227.855069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.230086] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1232.684617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.868187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1234.226249] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1236.962790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.076691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.987732] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1241.774706] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.779469] Lustre: MGS: Client 488504e5-44f1-435b-b27a-9d868565e099 (at 0@lo) reconnecting [ 1241.871555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.900099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1243.212708] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1252.320203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1252.324106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.331316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.346643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.347126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1253.351360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.973784] Lustre: server umount lustre-OST0000 complete [ 1258.464880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.465030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1258.465475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.465478] Lustre: Skipped 1 previous similar message [ 1258.468571] Lustre: Skipped 1 previous similar message [ 1263.584120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.586519] Lustre: Skipped 2 previous similar messages [ 1264.319596] Lustre: server umount lustre-MDT0000 complete [ 1265.610432] Lustre: server umount lustre-MDT0001 complete [ 1266.897456] Lustre: server umount MGS complete [ 1270.310330] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:06:00 (1788354360) [ 1278.876951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1281.754737] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1292.797101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.062958] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1297.775920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.858550] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1297.875648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1297.909922] Lustre: lustre-MDT0001: new disk, initializing [ 1297.934584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.943589] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1297.946196] Lustre: 93373:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1297.948909] LustreError: 93373:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1299.160237] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1303.274156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.362249] Lustre: lustre-OST0000: new disk, initializing [ 1303.363942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1303.366363] Lustre: 94140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1303.385974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.146762] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1306.720154] Lustre: 94567:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000043a217a5 x1875225412141824/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788354412 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1308.797243] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1310.872253] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.876559] Lustre: MGS: Client 2f0cedb1-d305-4239-bd6c-9546746ae3a1 (at 0@lo) reconnecting [ 1310.891053] Lustre: 95186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1310.897753] LustreError: 93359:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1310.901279] LustreError: 93359:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1310.910804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1310.943566] Lustre: lustre-MDT0000: new disk, initializing [ 1310.988146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1312.031724] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1312.031968] Lustre: 95202:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000b09a720 x1875225412147712/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788354418 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1312.039941] Lustre: 95202:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1312.165614] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1314.082090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1314.086378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1314.089907] Lustre: Skipped 1 previous similar message [ 1314.096711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1336.287726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.290621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.294728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.481985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.483646] Lustre: Skipped 4 previous similar messages [ 1350.111094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1350.158394] Lustre: server umount lustre-OST0000 complete [ 1354.720352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1354.720865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.722752] LustreError: Skipped 1 previous similar message [ 1354.727591] Lustre: Skipped 2 previous similar messages [ 1357.445857] Lustre: server umount lustre-MDT0000 complete [ 1373.151155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1373.235541] Lustre: server umount lustre-MDT0001 complete [ 1374.401512] Lustre: server umount MGS complete [ 1377.684520] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:07:48 (1788354468) [ 1383.122333] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1384.079700] Key type lgssc unregistered [ 1384.201309] LNet: 97710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.204400] LNetError: 97710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.214269] LNet: Removed LNI 192.168.203.142@tcp [ 1384.478585] Key type .llcrypt unregistered [ 1384.479830] Key type ._llcrypt unregistered [ 1390.894229] Key type ._llcrypt registered [ 1390.895327] Key type .llcrypt registered [ 1390.932201] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1391.225187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.283806] alg: No test for adler32 (adler32-zlib) [ 1392.135486] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1392.220719] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1393.799218] Key type lgssc registered [ 1394.156118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.728790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.737387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.879274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.228428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1401.878243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.993573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.230539] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1405.525900] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1407.188897] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1409.288238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.394694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.154733] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1413.385534] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.718327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1427.936373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1427.939057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.943151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.961346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.961740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.962765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.969527] Lustre: Skipped 1 previous similar message [ 1432.268108] Lustre: server umount lustre-OST0000 complete [ 1434.080565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.081261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.084026] Lustre: Skipped 1 previous similar message [ 1439.200085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.202078] Lustre: Skipped 2 previous similar messages [ 1439.570577] Lustre: server umount lustre-MDT0000 complete [ 1440.716071] LustreError: 98889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354530 with bad export cookie 14828913794311275171 [ 1440.718602] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1440.719819] LustreError: 98889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1440.827106] Lustre: server umount lustre-MDT0001 complete [ 1444.139250] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1445.185205] Key type lgssc unregistered [ 1445.307326] LNet: 102462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.310341] LNetError: 102462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.320273] LNet: Removed LNI 192.168.203.142@tcp [ 1445.595432] Key type .llcrypt unregistered [ 1445.596473] Key type ._llcrypt unregistered [ 1451.451163] Key type ._llcrypt registered [ 1451.452463] Key type .llcrypt registered [ 1451.485488] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1451.861972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.872698] alg: No test for adler32 (adler32-zlib) [ 1452.713316] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1452.791956] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1454.367142] Key type lgssc registered [ 1454.683583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.577750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1456.577776] LustreError: 103522:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1456.581135] Lustre: Skipped 1 previous similar message [ 1456.586190] LustreError: 103521:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1456.590070] LustreError: 103522:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1456.592761] LustreError: 103522:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1456.596551] LustreError: 103522:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1456.597483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.758695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.188843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.273446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.539949] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1463.085633] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing lsmod [ 1463.995475] LustreError: 104727:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1463.995519] LustreError: 104726:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1464.000270] LustreError: 104727:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1464.002557] LustreError: 104727:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1464.005372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.007377] LustreError: 104727:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1464.127732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.312097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1465.314942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.319879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.409131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.410664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.411372] Lustre: Skipped 1 previous similar message [ 1469.415136] Lustre: Skipped 1 previous similar message [ 1470.437105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.439469] Lustre: Skipped 2 previous similar messages [ 1471.130225] Lustre: server umount lustre-MDT0000 complete [ 1472.354384] LustreError: 103538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354562 with bad export cookie 3613342740072283922 [ 1472.355377] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.358691] LustreError: 103538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1472.364709] Lustre: 103551:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1472.461259] LustreError: 103991:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c414a93 x1875225652247936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1472.515345] Lustre: server umount lustre-MDT0001 complete [ 1483.804276] Lustre: server umount lustre-OST0000 complete [ 1486.447305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.587939] LustreError: 105790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.594423] LustreError: 105790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1486.608535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.818448] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1490.496661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.614668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.886805] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1494.060868] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.662589] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1497.738956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.677738] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1501.890309] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1512.166820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1515.487806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.491425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.495625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.538995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.539814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1519.825144] Lustre: server umount lustre-OST0000 complete [ 1521.631927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1521.635038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.640054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.642093] Lustre: Skipped 2 previous similar messages [ 1527.191685] Lustre: server umount lustre-MDT0000 complete [ 1527.775907] LustreError: 105791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.781138] LustreError: 105791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1528.444256] LustreError: 105771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354618 with bad export cookie 3613342740072284601 [ 1528.447711] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.449250] LustreError: 105771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1528.565598] Lustre: server umount lustre-MDT0001 complete [ 1532.227253] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1533.296493] Key type lgssc unregistered [ 1533.422298] LNet: 109338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.425269] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.435341] LNet: Removed LNI 192.168.203.142@tcp [ 1533.752291] Key type .llcrypt unregistered [ 1533.753563] Key type ._llcrypt unregistered [ 1543.088017] Key type ._llcrypt registered [ 1543.089145] Key type .llcrypt registered [ 1543.129111] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1543.570410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.587801] alg: No test for adler32 (adler32-zlib) [ 1544.450688] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1544.546650] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1546.135215] Key type lgssc registered [ 1546.559666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1549.299369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1549.305757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.479955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.920357] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1554.922339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.068785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.403077] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1558.661068] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1560.360339] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1562.461241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1562.570826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.447704] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1566.763648] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1574.887608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 1579.304097] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1580.031245] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1581.024630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.027981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.032046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.121032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.121763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1585.122735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.012212] Lustre: server umount lustre-OST0000 complete [ 1590.240882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.241601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.247446] Lustre: Skipped 1 previous similar message [ 1590.250063] Lustre: Skipped 1 previous similar message [ 1594.268411] Lustre: server umount lustre-MDT0000 complete [ 1595.360098] LustreError: 110730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.366737] LustreError: 110730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1595.536268] LustreError: 110711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354685 with bad export cookie 13984963298276258456 [ 1595.538838] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.541830] LustreError: 110711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1595.652456] Lustre: server umount lustre-MDT0001 complete [ 1599.265046] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1600.283909] Key type lgssc unregistered [ 1600.420339] LNet: 114382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.422745] LNetError: 114382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.433283] LNet: Removed LNI 192.168.203.142@tcp [ 1600.738688] Key type .llcrypt unregistered [ 1600.740227] Key type ._llcrypt unregistered [ 1607.676748] Key type ._llcrypt registered [ 1607.677945] Key type .llcrypt registered [ 1607.716392] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1608.019461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.027084] alg: No test for adler32 (adler32-zlib) [ 1608.873095] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1608.957911] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1610.535144] Key type lgssc registered [ 1610.910946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.387296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1613.392865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.531565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.884353] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1618.501069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.611165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1619.934865] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1622.318847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1624.050497] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1626.282196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.398995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.178854] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1630.423524] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1640.742663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1643.999784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.003570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.010180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.048349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.048949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.050603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.423329] Lustre: server umount lustre-OST0000 complete [ 1651.168768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.169307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.173069] Lustre: Skipped 1 previous similar message [ 1651.175975] Lustre: Skipped 2 previous similar messages [ 1656.288371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.290880] Lustre: Skipped 1 previous similar message [ 1656.725785] Lustre: server umount lustre-MDT0000 complete [ 1657.899167] LustreError: 115574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354748 with bad export cookie 8265563332571374686 [ 1657.902415] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.903678] LustreError: 115574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1658.003390] Lustre: server umount lustre-MDT0001 complete [ 1661.117971] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1661.990616] Key type lgssc unregistered [ 1662.107375] LNet: 119146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.109886] LNetError: 119146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.123304] LNet: Removed LNI 192.168.203.142@tcp [ 1662.391600] Key type .llcrypt unregistered [ 1662.392716] Key type ._llcrypt unregistered [ 1670.651427] Key type ._llcrypt registered [ 1670.652535] Key type .llcrypt registered [ 1670.697148] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1671.078703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.121100] alg: No test for adler32 (adler32-zlib) [ 1671.966832] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1672.046352] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1673.631289] Key type lgssc registered [ 1673.965534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.365801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.371196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.508876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.838278] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1681.477868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.598428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.947746] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1685.231309] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1686.984580] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1689.069321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.171852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.949777] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1693.094500] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1703.526138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 1710.781760] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1711.889625] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1711.891632] Lustre: Skipped 2 previous similar messages [ 1713.151704] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1713.154016] Lustre: Skipped 2 previous similar messages [ 1716.215928] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1716.227250] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1719.221087] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.42@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1719.224797] Lustre: Skipped 2 previous similar messages [ 1726.454327] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.42@tcp 192.168.0.[2,10]@tcp [ 1728.374536] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.42/24@tcp in log lustre-MDT0000 [ 1728.377681] Lustre: Skipped 2 previous similar messages [ 1736.693191] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.42/24@tcp [ 1736.696186] Lustre: Skipped 1 previous similar message [ 1739.745252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.747048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.747310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1739.749732] Lustre: Skipped 1 previous similar message [ 1744.865619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.867876] Lustre: Skipped 2 previous similar messages [ 1745.106805] Lustre: server umount lustre-OST0000 complete [ 1748.447823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1748.450224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.454320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1749.985041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.531543] Lustre: server umount lustre-MDT0000 complete [ 1753.755251] LustreError: 120564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354843 with bad export cookie 11056381788216887498 [ 1753.758717] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.760583] LustreError: 120564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1753.884983] Lustre: server umount lustre-MDT0001 complete [ 1757.526322] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1758.540442] Key type lgssc unregistered [ 1758.675353] LNet: 125876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.677953] LNetError: 125876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.689312] LNet: Removed LNI 192.168.203.142@tcp [ 1758.979405] Key type .llcrypt unregistered [ 1758.980479] Key type ._llcrypt unregistered [ 1769.608149] Key type ._llcrypt registered [ 1769.609491] Key type .llcrypt registered [ 1769.650093] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1769.972923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.002073] alg: No test for adler32 (adler32-zlib) [ 1770.858281] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1770.953705] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1772.535109] Key type lgssc registered [ 1772.900929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.567816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.576535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.746886] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1776.749511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.757573] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.42/24@tcp [ 1778.191941] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1781.050739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.165215] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1781.173031] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.42/24@tcp [ 1781.188135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.558844] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1784.970632] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1786.793676] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1789.059839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1789.183243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.069076] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1793.487695] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1801.513050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 1811.940171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.944374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1811.944648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.949020] Lustre: Skipped 1 previous similar message [ 1813.262609] Lustre: server umount lustre-OST0000 complete [ 1817.056537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1817.056939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.059962] LustreError: Skipped 1 previous similar message [ 1817.063490] Lustre: Skipped 1 previous similar message [ 1817.064095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1817.069642] Lustre: Skipped 1 previous similar message [ 1820.573378] Lustre: server umount lustre-MDT0000 complete [ 1821.795489] LustreError: 127394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354911 with bad export cookie 3447218257335036560 [ 1821.798512] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.798884] LustreError: 127394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1821.925045] Lustre: server umount lustre-MDT0001 complete [ 1825.695496] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1826.758957] Key type lgssc unregistered [ 1826.882331] LNet: 131018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.884711] LNetError: 131018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.893317] LNet: Removed LNI 192.168.203.142@tcp [ 1827.208743] Key type .llcrypt unregistered [ 1827.210246] Key type ._llcrypt unregistered [ 1835.670195] Key type ._llcrypt registered [ 1835.671239] Key type .llcrypt registered [ 1835.705684] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1841.858250] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1842.184678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1842.231350] alg: No test for adler32 (adler32-zlib) [ 1843.077520] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 1843.165433] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1844.743150] Key type lgssc registered [ 1845.153907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1856.096865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1856.103641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.186123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1857.201071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1857.234615] Lustre: lustre-MDT0000: new disk, initializing [ 1857.255796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.262463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1858.600937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1862.732321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.776071] Lustre: 134388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.803552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1862.805760] Lustre: Skipped 1 previous similar message [ 1862.839648] Lustre: lustre-MDT0001: new disk, initializing [ 1862.872430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.884124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1862.888425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1864.187854] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1867.870659] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1869.810507] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1872.013811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1872.109341] Lustre: lustre-OST0000: new disk, initializing [ 1872.111216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1872.113827] Lustre: 136356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1872.142868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.381368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1872.384830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1872.416178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1873.946707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1877.572983] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1878.862703] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1878.937445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1880.280623] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1880.346754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1882.594572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.594826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.601121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1882.603440] LustreError: Skipped 1 previous similar message [ 1887.714414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1887.716514] Lustre: Skipped 2 previous similar messages [ 1892.835444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.838207] Lustre: Skipped 1 previous similar message [ 1895.391145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1895.446412] Lustre: server umount lustre-OST0000 complete [ 1897.952966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.953632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.957676] Lustre: Skipped 2 previous similar messages [ 1897.960833] Lustre: Skipped 1 previous similar message [ 1902.875834] Lustre: server umount lustre-MDT0000 complete [ 1904.112237] LustreError: 135183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788354994 with bad export cookie 18341472346878459453 [ 1904.115657] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.115762] LustreError: 135183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.250873] Lustre: server umount lustre-MDT0001 complete [ 1906.931965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.112080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.388905] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1911.154931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.273656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1912.606938] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1914.969264] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1916.722296] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1918.909475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1920.782574] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1923.079181] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.382206] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1932.591873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1933.906840] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1933.982965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1937.284017] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.343979] Lustre: lustre-OST0001: new disk, initializing [ 1937.345858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1937.348364] Lustre: 141995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1937.378948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1937.381693] Lustre: Skipped 1 previous similar message [ 1939.240591] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1942.440468] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1945.084543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1945.088698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1945.112899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1945.568773] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1946.979969] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1947.073195] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1948.890333] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92dc078ad000.ost_server_uuid 50 [ 1949.428781] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92dc078ad000.ost_server_uuid in FULL state after 0 sec [ 1955.299072] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.299541] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.300787] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1955.304052] Lustre: Skipped 1 previous similar message [ 1955.306947] LustreError: Skipped 1 previous similar message [ 1957.439862] Lustre: server umount lustre-OST0001 complete [ 1960.419067] LustreError: 140801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.419588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.424316] LustreError: 140801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1960.428529] Lustre: Skipped 1 previous similar message [ 1965.538259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.538439] LustreError: 140389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.540941] Lustre: Skipped 4 previous similar messages [ 1965.546837] LustreError: 140389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1970.657900] LustreError: 140390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.665034] LustreError: 140390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1973.215155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1973.270773] Lustre: server umount lustre-OST0000 complete [ 1975.777300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.692265] Lustre: server umount lustre-MDT0000 complete [ 1980.895856] LustreError: 138570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.901277] LustreError: 138570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1981.946315] LustreError: 141994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355072 with bad export cookie 18341472346878460321 [ 1981.948079] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1981.959987] LustreError: 141994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1982.086975] Lustre: server umount lustre-MDT0001 complete [ 1988.771834] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1989.847048] Key type lgssc unregistered [ 1989.977364] LNet: 145017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.979667] LNetError: 145017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.987408] LNet: Removed LNI 192.168.203.142@tcp [ 1990.332649] Key type .llcrypt unregistered [ 1990.334416] Key type ._llcrypt unregistered [ 2003.801299] Key type ._llcrypt registered [ 2003.802493] Key type .llcrypt registered [ 2003.841200] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2009.986702] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2010.335847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.357962] alg: No test for adler32 (adler32-zlib) [ 2011.217166] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 2011.315122] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2012.903189] Key type lgssc registered [ 2013.334349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.216501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2024.223996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.322987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2025.333842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2025.367366] Lustre: lustre-MDT0000: new disk, initializing [ 2025.385653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.391197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2026.712723] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2030.759434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.808554] Lustre: 149141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.839769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2030.842208] Lustre: Skipped 1 previous similar message [ 2030.883205] Lustre: lustre-MDT0001: new disk, initializing [ 2030.924465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.938714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2030.942564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2032.228595] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2035.743905] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.347405] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2039.392787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2039.479606] Lustre: lustre-OST0000: new disk, initializing [ 2039.481189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2039.483354] Lustre: 151109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2039.510100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.267652] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2041.849275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2041.852972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2041.881133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2045.001535] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2046.987320] Lustre: Failing over lustre-OST0000 [ 2047.052610] Lustre: server umount lustre-OST0000 complete [ 2059.952097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2060.071354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.078615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.086909] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2061.789378] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2062.040894] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2063.276251] Lustre: Failing over lustre-MDT0000 [ 2063.368404] Lustre: server umount lustre-MDT0000 complete [ 2065.377076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.380650] Lustre: Skipped 3 previous similar messages [ 2068.447153] Lustre: 147371:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788355142/real 1788355142] req@00000000736ce832 x1875226238413312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788355158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.496553] LustreError: 149153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.501747] LustreError: 149153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2072.020359] LustreError: 151907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.503099] Lustre: 147373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788355147/real 1788355147] req@00000000c2a74e28 x1875226238415104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788355163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.615968] LustreError: 149152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.620470] LustreError: 149152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2075.953649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.985101] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.989105] Lustre: 149145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2076.054063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2076.069497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.095200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2077.140515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2077.232889] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2081.254576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2081.261394] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2081.265171] Lustre: Skipped 1 previous similar message [ 2081.283627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2086.370770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2086.371032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.373629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.376821] Lustre: Skipped 1 previous similar message [ 2088.332886] Lustre: server umount lustre-OST0000 complete [ 2091.488696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2091.489087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.489854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.489858] Lustre: Skipped 1 previous similar message [ 2091.491751] LustreError: Skipped 1 previous similar message [ 2091.500523] Lustre: Skipped 1 previous similar message [ 2095.704254] Lustre: server umount lustre-MDT0000 complete [ 2096.608069] LustreError: 149944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.614254] LustreError: 149944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2096.973291] LustreError: 149134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355187 with bad export cookie 4173882477580068896 [ 2096.976439] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.978644] LustreError: 149134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2097.112685] Lustre: server umount lustre-MDT0001 complete [ 2100.584674] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2101.491863] Key type lgssc unregistered [ 2101.609354] LNet: 154757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.612774] LNetError: 154757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.621338] LNet: Removed LNI 192.168.203.142@tcp [ 2101.910231] Key type .llcrypt unregistered [ 2101.911282] Key type ._llcrypt unregistered [ 2121.532657] Key type ._llcrypt registered [ 2121.533778] Key type .llcrypt registered [ 2121.574210] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2121.887228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.953946] alg: No test for adler32 (adler32-zlib) [ 2122.807819] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 2122.894675] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2124.471118] Key type lgssc registered [ 2124.815753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2127.311603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2127.318690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.466280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.853171] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2133.665291] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.692884] LustreError: 156305:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2135.697093] LustreError: 156305:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2135.701195] LustreError: 156305:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2135.828044] Lustre: server umount lustre-MDT0000 complete [ 2140.875816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2144.081259] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2156.134829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.214763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2156.224985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2156.257768] Lustre: lustre-MDT0000: new disk, initializing [ 2156.277383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.283091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2157.557020] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2161.491395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.529872] Lustre: 159602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2161.552725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2161.555837] Lustre: Skipped 1 previous similar message [ 2161.588310] Lustre: lustre-MDT0001: new disk, initializing [ 2161.617128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.628200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2161.633118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2162.805089] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2166.326258] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.966908] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2170.044157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2170.139660] Lustre: lustre-OST0000: new disk, initializing [ 2170.141303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2170.143526] Lustre: 161568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2170.169650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.855785] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2175.427376] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.490238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2175.493612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2175.530641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2177.154160] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2177.656285] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:21:08 (1788355268) [ 2236.895612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.895831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.899980] Lustre: Skipped 2 previous similar messages [ 2236.904545] Lustre: Skipped 2 previous similar messages [ 2242.015666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.018960] Lustre: Skipped 2 previous similar messages [ 2242.914147] Lustre: server umount lustre-MDT0000 complete [ 2244.139540] LustreError: 159593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355334 with bad export cookie 15816266266477633258 [ 2244.140661] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.145632] LustreError: 159593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2247.135496] LustreError: 162270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.135523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.141125] LustreError: 162270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2247.145099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2249.183458] LustreError: 159608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.183466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2254.303299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2254.303299] LustreError: 162270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.399073] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.469851] Lustre: server umount lustre-MDT0001 complete [ 2274.271134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2274.323557] Lustre: server umount lustre-OST0000 complete [ 2277.265215] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2280.388875] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2292.138447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.215833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2292.226859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2292.265713] Lustre: lustre-MDT0000: new disk, initializing [ 2292.288768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.295521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2293.634668] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2297.957867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.999526] Lustre: 166192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2298.020183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2298.021916] Lustre: Skipped 1 previous similar message [ 2298.057670] Lustre: lustre-MDT0001: new disk, initializing [ 2298.101328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2298.104975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2299.393389] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2302.980515] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.765906] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.906405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.996805] Lustre: lustre-OST0000: new disk, initializing [ 2306.998387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2307.000707] Lustre: 168159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2307.028604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.030733] Lustre: Skipped 1 previous similar message [ 2307.640358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2307.644590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2307.677671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2309.336171] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2313.339054] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2314.644432] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2314.712613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2316.095742] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2316.168234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2317.794085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.794558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2317.795408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2317.799064] Lustre: Skipped 1 previous similar message [ 2317.801146] LustreError: Skipped 1 previous similar message [ 2331.103109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2331.157912] Lustre: server umount lustre-OST0000 complete [ 2333.153515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.157090] Lustre: Skipped 1 previous similar message [ 2338.272589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.274522] Lustre: Skipped 8 previous similar messages [ 2338.589877] Lustre: server umount lustre-MDT0000 complete [ 2339.773265] LustreError: 166185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355429 with bad export cookie 15816266266477792074 [ 2339.776377] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.777283] LustreError: 166185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2339.900824] Lustre: server umount lustre-MDT0001 complete [ 2342.653783] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:23:52 (1788355432) [ 2347.956086] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2351.155428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2363.107071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.209516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2363.222914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2363.264373] Lustre: lustre-MDT0000: new disk, initializing [ 2363.288131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.295658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2364.658695] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2368.903525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.957800] Lustre: 172907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2369.083276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2370.491389] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2374.233623] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2376.134445] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2378.335400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2378.448873] Lustre: 174884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2380.376851] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2384.194326] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2387.447319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2387.482088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2392.545494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2392.546193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.547899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.549152] LustreError: Skipped 1 previous similar message [ 2392.554338] Lustre: Skipped 1 previous similar message [ 2393.298753] Lustre: server umount lustre-OST0000 complete [ 2394.591776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.700473] Lustre: server umount lustre-MDT0000 complete [ 2397.665310] LustreError: 172918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.673173] LustreError: 172918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2397.945263] LustreError: 172901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355488 with bad export cookie 15816266266477792942 [ 2397.949431] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.951027] LustreError: 172901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2398.072992] Lustre: server umount lustre-MDT0001 complete [ 2400.831695] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:24:51 (1788355491) [ 2406.288213] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2409.435446] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2421.032213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.114091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2421.124590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2421.127294] Lustre: Skipped 3 previous similar messages [ 2421.169660] Lustre: lustre-MDT0000: new disk, initializing [ 2421.172033] Lustre: Skipped 2 previous similar messages [ 2421.194344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.196304] Lustre: Skipped 2 previous similar messages [ 2421.201232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2421.203965] Lustre: Skipped 2 previous similar messages [ 2422.584650] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2426.790737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.834203] Lustre: 179196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2426.929814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2428.285939] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2432.022043] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2433.795742] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2436.008921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2436.121843] Lustre: 181171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2438.119620] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2440.215488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2441.889207] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.281134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.281338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.283767] LustreError: Skipped 1 previous similar message [ 2445.289241] Lustre: Skipped 3 previous similar messages [ 2449.745294] Lustre: server umount lustre-OST0000 complete [ 2452.448015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2460.640198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.642146] Lustre: Skipped 8 previous similar messages [ 2465.247159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2465.307164] Lustre: server umount lustre-MDT0000 complete [ 2465.759971] LustreError: 179206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.764140] LustreError: 179206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2466.474168] LustreError: 179189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355556 with bad export cookie 15816266266477794013 [ 2466.476835] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.478634] LustreError: 179189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2466.608848] Lustre: server umount lustre-MDT0001 complete [ 2470.147692] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2471.094199] Key type lgssc unregistered [ 2471.219370] LNet: 183024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.222239] LNetError: 183024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.234302] LNet: Removed LNI 192.168.203.142@tcp [ 2471.532059] Key type .llcrypt unregistered [ 2471.533602] Key type ._llcrypt unregistered [ 2480.499573] Key type ._llcrypt registered [ 2480.501412] Key type .llcrypt registered [ 2480.540129] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2480.911455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.963156] alg: No test for adler32 (adler32-zlib) [ 2481.807956] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 2481.897869] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2483.471139] Key type lgssc registered [ 2483.796884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.201168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.206172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.325511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.586863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2491.141034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.272246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.591089] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2494.942425] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.688436] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2498.827716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.936326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.758868] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2503.070654] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2512.230912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2526.688142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2526.690328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.694356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.712774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.713581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2527.713838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.714838] Lustre: Skipped 1 previous similar message [ 2530.895863] Lustre: server umount lustre-OST0000 complete [ 2532.319928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2532.322926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.327772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.952676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.955301] Lustre: Skipped 2 previous similar messages [ 2538.272725] Lustre: server umount lustre-MDT0000 complete [ 2539.496963] LustreError: 184398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355629 with bad export cookie 17023714338359082133 [ 2539.500443] LustreError: 184398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2539.500479] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.620319] Lustre: server umount lustre-MDT0001 complete [ 2543.342401] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2544.471001] Key type lgssc unregistered [ 2544.616560] LNet: 187969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.620661] LNetError: 187969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.631489] LNet: Removed LNI 192.168.203.142@tcp [ 2544.949150] Key type .llcrypt unregistered [ 2544.950713] Key type ._llcrypt unregistered [ 2554.354493] Key type ._llcrypt registered [ 2554.355699] Key type .llcrypt registered [ 2554.391805] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2554.817050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2554.856127] alg: No test for adler32 (adler32-zlib) [ 2555.721195] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 2555.821247] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2557.407129] Key type lgssc registered [ 2557.872975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2560.462248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2560.467649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.609208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.922451] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2565.780749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.908457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2567.246129] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2569.644469] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2571.426789] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2573.616525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.747954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.585274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2577.934600] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2586.022474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2591.201364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.202847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.204433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.207344] Lustre: Skipped 1 previous similar message [ 2593.756620] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 2593.760385] Lustre: Skipped 1 previous similar message [ 2596.322368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.324210] Lustre: Skipped 1 previous similar message [ 2598.869343] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 2603.989679] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 2603.992755] Lustre: Skipped 2 previous similar messages [ 2605.023053] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2605.072313] Lustre: server umount lustre-OST0000 complete [ 2607.400136] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92dc05d00000.ost_server_uuid 50 [ 2607.894490] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92dc05d00000.ost_server_uuid in DISCONN state after 0 sec [ 2608.418995] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2621.920765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.921246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.924166] Lustre: Skipped 1 previous similar message [ 2624.477888] Lustre: server umount lustre-MDT0000 complete [ 2625.752336] LustreError: 189353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355715 with bad export cookie 4346152564412890917 [ 2625.755790] LustreError: 189353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2625.756367] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.888674] Lustre: server umount lustre-MDT0001 complete [ 2628.600382] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:28:38 (1788355718) [ 2630.734881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.912161] LustreError: 193336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.917435] LustreError: 193336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2630.937218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.192271] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2634.902970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.040748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2636.425623] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2638.846455] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2640.696170] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2642.924944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.059634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.904191] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2647.304348] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2649.513069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.586964] Lustre: lustre-OST0001: new disk, initializing [ 2649.588600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2649.590993] Lustre: 196174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2651.586415] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2655.233063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2655.237841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2655.267591] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2655.267717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2657.122174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.123856] Lustre: Skipped 1 previous similar message [ 2657.128396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.202730] Lustre: server umount lustre-OST0000 complete [ 2662.373590] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.378139] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2664.236361] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2664.314266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2665.625922] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2665.697035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2665.941781] LustreError: 195543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.061913] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.067893] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2675.680171] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.680220] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.683260] LustreError: Skipped 1 previous similar message [ 2677.729646] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.734628] LustreError: 195157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2681.808513] Lustre: server umount lustre-OST0001 complete [ 2686.432072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2686.436434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.443329] Lustre: Skipped 1 previous similar message [ 2689.117590] Lustre: server umount lustre-MDT0000 complete [ 2690.369384] LustreError: 193317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355780 with bad export cookie 4346152564412891974 [ 2690.373803] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.491868] Lustre: server umount lustre-MDT0001 complete [ 2698.040971] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:29:48 (1788355788) [ 2700.117553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.208192] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2700.218960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2700.288230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.290292] Lustre: Skipped 1 previous similar message [ 2701.564168] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2704.218832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.248382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2704.259656] Lustre: 199590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2705.600364] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2707.811372] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.598797] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2711.706267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.769248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2713.421259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2713.615598] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2715.884368] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.045094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.069586] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2720.015172] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2721.304681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2722.249955] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2723.808141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2723.808206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.810302] LustreError: Skipped 1 previous similar message [ 2723.810725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2723.813663] Lustre: Skipped 2 previous similar messages [ 2723.815012] Lustre: Skipped 9 previous similar messages [ 2729.082962] Lustre: server umount lustre-OST0000 complete [ 2731.490712] LustreError: 201436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.495020] LustreError: 201436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2741.729112] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.730645] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.731277] LustreError: Skipped 1 previous similar message [ 2745.609957] Lustre: server umount lustre-OST0001 complete [ 2761.183155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2761.283723] Lustre: server umount lustre-MDT0000 complete [ 2762.946159] LustreError: 199580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355853 with bad export cookie 4346152564412893136 [ 2762.950410] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.953925] LustreError: 199580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2763.136927] Lustre: server umount lustre-MDT0001 complete [ 2776.686814] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:31:06 (1788355866) [ 2786.475038] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2790.060431] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2802.091128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.169114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2802.180824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2802.218556] Lustre: lustre-MDT0000: new disk, initializing [ 2802.240595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.243571] Lustre: Skipped 3 previous similar messages [ 2802.249756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2803.637555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2807.862054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.906584] Lustre: 207652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2807.929523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2807.932423] Lustre: Skipped 1 previous similar message [ 2807.967479] Lustre: lustre-MDT0001: new disk, initializing [ 2808.017480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2808.021637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2809.437908] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2812.923450] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2814.785277] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2816.958147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.058775] Lustre: lustre-OST0000: new disk, initializing [ 2817.061217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2817.065167] Lustre: 209617:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2818.973903] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2822.755402] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2823.175161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2823.180051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2823.225099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2824.238904] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2824.312887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2825.754518] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2825.836335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2828.257847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.260160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.260596] LustreError: Skipped 1 previous similar message [ 2828.264441] Lustre: Skipped 3 previous similar messages [ 2828.265234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2828.270088] Lustre: Skipped 15 previous similar messages [ 2832.656236] Lustre: server umount lustre-OST0000 complete [ 2848.223111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2848.289642] Lustre: server umount lustre-MDT0000 complete [ 2848.736674] LustreError: 207662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.742181] LustreError: 207662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2849.539040] LustreError: 207644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788355939 with bad export cookie 4346152564412894900 [ 2849.542589] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.544956] LustreError: 207644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2849.675286] Lustre: server umount lustre-MDT0001 complete [ 2852.424732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.958122] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2856.696692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.215074] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2860.684215] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2862.574277] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2865.053573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.206936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2869.737239] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2871.378210] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2877.579402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2879.026682] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2879.102792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2881.503976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.838050] Lustre: server umount lustre-OST0000 complete [ 2887.752570] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2889.078133] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2912.639238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.741093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.744830] Lustre: Skipped 5 previous similar messages [ 2914.711193] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2917.319413] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2917.928709] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2921.110868] Lustre: server umount lustre-OST0000 complete [ 2923.487850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2923.490712] LustreError: Skipped 1 previous similar message [ 2923.492333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2923.496815] Lustre: Skipped 5 previous similar messages [ 2924.693061] Lustre: server umount lustre-MDT0000 complete [ 2926.061640] LustreError: 211814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356016 with bad export cookie 4346152564412895782 [ 2926.065406] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.066403] LustreError: 211814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2926.191863] Lustre: server umount lustre-MDT0001 complete [ 2929.234896] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:33:39 (1788356019) [ 2931.697887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.874078] LustreError: 217832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.882057] LustreError: 217832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2933.377948] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2936.650510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.300525] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2940.885283] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2942.907479] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2945.372424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.583650] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2950.326235] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2951.987070] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2958.193986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2959.836724] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2959.921175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2962.487203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.565186] Lustre: lustre-OST0001: new disk, initializing [ 2962.568067] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2962.572726] Lustre: 221242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2964.730188] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2968.727515] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2970.300923] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2971.134801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2971.139079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2971.167851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2971.406112] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2972.987997] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2973.074047] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2976.233847] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2976.236499] Lustre: Skipped 9 previous similar messages [ 2979.906649] Lustre: server umount lustre-OST0001 complete [ 2982.044754] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2982.125636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2983.588274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2983.661794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2996.708312] LustreError: 219652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.715093] LustreError: 219652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3007.716540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.233593] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3013.096149] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3013.102477] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3013.102516] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3013.223471] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3013.861983] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3018.213940] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.218331] LustreError: Skipped 3 previous similar messages [ 3020.668615] Lustre: server umount lustre-OST0001 complete [ 3047.391245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3047.458859] Lustre: server umount lustre-OST0000 complete [ 3063.263130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3063.332316] Lustre: server umount lustre-MDT0000 complete [ 3064.665463] LustreError: 217812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356154 with bad export cookie 4346152564412896713 [ 3064.668194] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.670495] LustreError: 217812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3070.833558] Lustre: server umount lustre-MDT0001 complete [ 3079.299861] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:36:09 (1788356169) [ 3081.884367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.017738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3082.031612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3082.112972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.116809] Lustre: Skipped 5 previous similar messages [ 3083.699043] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3086.909542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.965049] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3086.977736] Lustre: 226494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3088.649316] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3091.422780] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.397867] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.774204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.837178] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3097.910898] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3100.252764] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3104.249902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3104.279637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3106.428565] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3109.189829] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3109.839676] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3111.362846] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3111.454388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3113.026507] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3113.111259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3115.110949] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92dc05332000.ost_server_uuid 50 [ 3115.704521] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92dc05332000.ost_server_uuid in FULL state after 0 sec [ 3116.106595] Lustre: Permanently deactivating lustre-OST0001 [ 3116.116496] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3117.104034] Lustre: Permanently reactivating lustre-OST0001 [ 3121.632478] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.638810] Lustre: Skipped 8 previous similar messages [ 3124.288170] Lustre: server umount lustre-OST0001 complete [ 3127.266883] LustreError: 228340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.274052] LustreError: 228340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 3131.729772] Lustre: server umount lustre-OST0000 complete [ 3139.220603] Lustre: server umount lustre-MDT0000 complete [ 3140.517187] LustreError: 226531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356230 with bad export cookie 4346152564412897987 [ 3140.519748] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.521544] LustreError: 226531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3140.640609] Lustre: server umount lustre-MDT0001 complete [ 3148.613126] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:37:18 (1788356238) [ 3151.144027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.233936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3151.244893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3151.250348] Lustre: Skipped 5 previous similar messages [ 3152.625850] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3155.531471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.565723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3155.577341] Lustre: 233175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3156.961483] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3159.380340] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3161.250885] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3163.583378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.676952] Lustre: Permanently deactivating lustre-OST0000 [ 3163.682562] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3165.882525] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3165.944149] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3165.978825] LustreError: 235419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3165.979573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3168.499210] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3170.771766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.739590] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3175.122905] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3175.701898] Lustre: Permanently reactivating lustre-OST0000 [ 3175.763943] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3175.804314] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3175.806632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3175.808692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3175.818970] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3175.820341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3175.823065] Lustre: Skipped 1 previous similar message [ 3190.755375] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3190.758848] LustreError: Skipped 4 previous similar messages [ 3194.111297] Lustre: server umount lustre-OST0001 complete [ 3201.627496] Lustre: server umount lustre-OST0000 complete [ 3218.399127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3218.762266] Lustre: server umount lustre-MDT0000 complete [ 3221.401797] LustreError: 233917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356311 with bad export cookie 4346152564412899618 [ 3221.414750] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.635060] Lustre: server umount lustre-MDT0001 complete [ 3229.027323] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3231.457654] Key type lgssc unregistered [ 3231.664857] LNet: 238041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.669455] LNetError: 238041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3232.742994] LNet: Removed LNI 192.168.203.142@tcp [ 3233.311893] Key type .llcrypt unregistered [ 3233.313749] Key type ._llcrypt unregistered [ 3249.348451] Key type ._llcrypt registered [ 3249.352485] Key type .llcrypt registered [ 3249.539752] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3250.906022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3250.941092] alg: No test for adler32 (adler32-zlib) [ 3252.027708] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3252.267951] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3253.919686] Key type lgssc registered [ 3255.820682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.510019] hrtimer: interrupt took 10055848 ns [ 3265.334385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3265.347569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.615482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.211666] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3274.030214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.106045] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3274.125572] Lustre: 239480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3274.135418] Lustre: Permanently deactivating lustre-MDT0001 [ 3274.145106] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3274.349602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3276.934538] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3277.948127] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3281.017419] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3284.108146] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3287.330757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.496164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.933221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3291.623365] LustreError: 241860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.629109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3291.634472] LustreError: 241860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3293.338275] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3296.739082] LustreError: 241860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.920390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.046597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.115294] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3299.117547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3299.128378] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3299.825780] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3303.583581] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3305.590562] Lustre: Permanently reactivating lustre-MDT0001 [ 3305.602424] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3305.605975] Lustre: Skipped 2 previous similar messages [ 3306.065490] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.072780] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3306.078786] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3306.087626] LustreError: 243053:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3306.090256] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3306.097256] LustreError: 243053:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3306.097496] LustreError: 243053:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -108 [ 3306.117192] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3306.117192] [ 3306.123761] LustreError: 239485:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3312.102775] Lustre: lustre-MDT0001: Client e13e302e-38df-4f1b-b44f-ac46419c00e4 (at 192.168.203.42@tcp) reconnecting [ 3315.851223] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3316.680741] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3319.330939] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3319.452977] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3321.613125] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3321.730436] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3322.996273] Lustre: Permanently deactivating lustre-MDT0001 [ 3323.008505] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3323.013141] Lustre: Skipped 2 previous similar messages [ 3329.043847] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3334.116770] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3334.122557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.133630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.160464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.162665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.166379] Lustre: Skipped 2 previous similar messages [ 3336.172076] Lustre: Skipped 2 previous similar messages [ 3337.929864] Lustre: server umount lustre-MDT0000 complete [ 3339.728890] LustreError: 239473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356429 with bad export cookie 17020395747550943869 [ 3339.731746] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.734829] LustreError: 239473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3339.899561] Lustre: server umount lustre-MDT0001 complete [ 3347.841442] Lustre: server umount lustre-OST0000 complete [ 3353.292050] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:40:43 (1788356443) [ 3358.425578] Lustre: server umount lustre-OST0001 complete [ 3361.631265] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3366.370618] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3385.488612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.622660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3385.627296] Lustre: Skipped 2 previous similar messages [ 3385.641458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3385.693769] Lustre: lustre-MDT0000: new disk, initializing [ 3385.738652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.751558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3388.096354] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3394.457641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.531436] Lustre: 247737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3394.567367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3394.570726] Lustre: Skipped 1 previous similar message [ 3394.624378] Lustre: lustre-MDT0001: new disk, initializing [ 3394.675766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.693622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3394.699356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3396.908369] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3402.445496] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3405.444614] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3408.949821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.081154] Lustre: lustre-OST0000: new disk, initializing [ 3409.083693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3409.088162] Lustre: 249706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3412.143365] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3412.685587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3412.690127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3412.744470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3417.103507] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3418.149841] LustreError: 247745:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3423.569285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3423.680315] Lustre: lustre-OST0001: new disk, initializing [ 3423.684653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3423.692029] Lustre: 250822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3423.744107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3423.749022] Lustre: Skipped 1 previous similar message [ 3426.654753] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3428.167125] LustreError: 247745:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3428.185715] LustreError: 248975:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3431.968320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3431.973567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3432.026917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3432.363543] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3433.206095] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3438.287300] LustreError: 248975:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3439.536154] Lustre: lustre-OST0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 3442.151327] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.151969] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3442.153255] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3442.161071] Lustre: Skipped 1 previous similar message [ 3442.168309] LustreError: Skipped 1 previous similar message [ 3445.475136] Lustre: server umount lustre-OST0001 complete [ 3447.268873] LustreError: 249692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3447.283153] LustreError: 249692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3452.388637] LustreError: 250127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.399772] LustreError: 250127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3457.508391] LustreError: 249691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.523125] LustreError: 249691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3462.626171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.631235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.639517] Lustre: Skipped 1 previous similar message [ 3462.644214] Lustre: Skipped 2 previous similar messages [ 3465.519972] Lustre: server umount lustre-OST0000 complete [ 3472.865175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.868842] Lustre: Skipped 4 previous similar messages [ 3473.859047] Lustre: server umount lustre-MDT0000 complete [ 3475.900971] LustreError: 247730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356566 with bad export cookie 17020395747550946690 [ 3475.905300] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.908197] LustreError: 247730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3490.271530] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3490.398224] Lustre: server umount lustre-MDT0001 complete [ 3495.578544] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3497.060884] Key type lgssc unregistered [ 3497.245890] LNet: 252861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3497.250491] LNetError: 252861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3497.262692] LNet: Removed LNI 192.168.203.142@tcp [ 3497.733126] Key type .llcrypt unregistered [ 3497.735251] Key type ._llcrypt unregistered [ 3517.619162] Key type ._llcrypt registered [ 3517.620899] Key type .llcrypt registered [ 3517.673558] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3518.158309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.190840] alg: No test for adler32 (adler32-zlib) [ 3519.097855] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3519.223113] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3520.823367] Key type lgssc registered [ 3521.419685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3525.562725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3525.576607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.721356] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3526.742728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3526.831905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.973997] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3533.096170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.177045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3533.193347] Lustre: 255045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3533.337956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3535.374870] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3538.993649] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3541.770387] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3545.210348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.317873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3545.413805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.305667] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3552.106642] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3553.324542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3594.212625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.213265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.219251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.225781] Lustre: Skipped 1 previous similar message [ 3599.331221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.335981] Lustre: Skipped 1 previous similar message [ 3599.739107] Lustre: server umount lustre-OST0000 complete [ 3601.654639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3606.778961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.957742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.455669] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3612.722697] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3624.356619] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3624.361890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3624.361911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3624.372045] Lustre: Skipped 1 previous similar message [ 3650.018463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.020080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.025810] LustreError: Skipped 2 previous similar messages [ 3650.036232] Lustre: Skipped 1 previous similar message [ 3650.041701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.045095] Lustre: Skipped 1 previous similar message [ 3652.923976] Lustre: server umount lustre-OST0000 complete [ 3655.137177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.151953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.160964] Lustre: Skipped 1 previous similar message [ 3660.256614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.260522] Lustre: Skipped 1 previous similar message [ 3661.032431] Lustre: server umount lustre-MDT0000 complete [ 3662.724516] LustreError: 255039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356752 with bad export cookie 3797265832173783450 [ 3662.730028] LustreError: 255039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3662.731093] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.882122] Lustre: server umount lustre-MDT0001 complete [ 3667.628520] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3668.933345] Key type lgssc unregistered [ 3669.102336] LNet: 260463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.106167] LNetError: 260463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3669.114513] LNet: Removed LNI 192.168.203.142@tcp [ 3669.535961] Key type .llcrypt unregistered [ 3669.538101] Key type ._llcrypt unregistered [ 3680.743545] Key type ._llcrypt registered [ 3680.745628] Key type .llcrypt registered [ 3680.800756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3681.202233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3681.254914] alg: No test for adler32 (adler32-zlib) [ 3682.161984] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3682.291954] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3683.911227] Key type lgssc registered [ 3684.407374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3687.943528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3687.954453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.172376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.007321] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3695.256432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.463825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3697.395228] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3700.802860] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3703.335973] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3706.374146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3706.539883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.160610] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3712.042875] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3714.730875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3735.520510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3735.527753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.534804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.546356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.546487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.552740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.753659] Lustre: server umount lustre-OST0000 complete [ 3741.665193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.665270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3741.665975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.665979] Lustre: Skipped 1 previous similar message [ 3741.670588] Lustre: Skipped 1 previous similar message [ 3745.311127] Lustre: server umount lustre-MDT0000 complete [ 3746.701230] LustreError: 261837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356836 with bad export cookie 3739616981730837760 [ 3746.703463] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.706254] LustreError: 261837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3746.826840] Lustre: server umount lustre-MDT0001 complete [ 3750.811063] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3751.931238] Key type lgssc unregistered [ 3752.072428] LNet: 266177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.075836] LNetError: 266177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3752.084800] LNet: Removed LNI 192.168.203.142@tcp [ 3752.433117] Key type .llcrypt unregistered [ 3752.434775] Key type ._llcrypt unregistered [ 3760.724978] Key type ._llcrypt registered [ 3760.726444] Key type .llcrypt registered [ 3760.780490] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3761.267839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3761.278862] alg: No test for adler32 (adler32-zlib) [ 3762.160457] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3762.284692] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3763.887416] Key type lgssc registered [ 3764.343412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.541763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3767.552349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.754744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3770.399030] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3773.726440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.902464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.717196] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3778.638805] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3780.877970] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3783.745918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3783.900959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.152853] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3788.916784] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.131606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3803.618288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.624649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.624950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.629691] Lustre: Skipped 1 previous similar message [ 3808.738860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.741547] Lustre: Skipped 1 previous similar message [ 3809.690401] Lustre: server umount lustre-OST0000 complete [ 3813.856456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.858557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.862048] Lustre: Skipped 2 previous similar messages [ 3817.307540] Lustre: server umount lustre-MDT0000 complete [ 3818.798658] LustreError: 267359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788356908 with bad export cookie 17707263868579831386 [ 3818.801648] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.804812] LustreError: 267359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3818.925633] Lustre: server umount lustre-MDT0001 complete [ 3823.369223] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3824.600950] Key type lgssc unregistered [ 3824.740622] LNet: 271073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3824.743971] LNetError: 271073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.758430] LNet: Removed LNI 192.168.203.142@tcp [ 3825.101619] Key type .llcrypt unregistered [ 3825.103351] Key type ._llcrypt unregistered [ 3836.251339] Key type ._llcrypt registered [ 3836.252369] Key type .llcrypt registered [ 3836.298910] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3836.648571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.659841] alg: No test for adler32 (adler32-zlib) [ 3837.557273] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3837.670240] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3839.287175] Key type lgssc registered [ 3839.794572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3843.097144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3843.109439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.310262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.085892] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3849.793403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.987978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.621788] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3854.689436] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3857.018221] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3859.851077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3860.000099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.457913] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3865.971857] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.224080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3881.439996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.443581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.454586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.514094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.514241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.526673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3889.633701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.636865] Lustre: Skipped 2 previous similar messages [ 3894.755318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.757565] Lustre: Skipped 1 previous similar message [ 3895.263133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3895.337299] Lustre: server umount lustre-OST0000 complete [ 3899.872771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.881928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3899.884912] Lustre: Skipped 1 previous similar message [ 3910.112449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.118778] Lustre: Skipped 3 previous similar messages [ 3911.136651] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3911.201267] Lustre: server umount lustre-MDT0000 complete [ 3912.729319] LustreError: 272447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357002 with bad export cookie 4443595335242078506 [ 3912.731826] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3912.736690] LustreError: 272447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3912.882535] Lustre: server umount lustre-MDT0001 complete [ 3917.284874] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3918.555980] Key type lgssc unregistered [ 3918.705471] LNet: 276771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3918.709527] LNetError: 276771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3918.718990] LNet: Removed LNI 192.168.203.142@tcp [ 3919.099231] Key type .llcrypt unregistered [ 3919.101148] Key type ._llcrypt unregistered [ 3927.471574] Key type ._llcrypt registered [ 3927.473102] Key type .llcrypt registered [ 3927.527392] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3928.048158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3928.055754] alg: No test for adler32 (adler32-zlib) [ 3928.943924] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 3929.058888] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3930.655341] Key type lgssc registered [ 3931.138696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3934.192037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3934.205620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.436723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.051964] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3940.296640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.482335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.097815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3944.897270] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.121784] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3950.080310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3950.259391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.664221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3955.768619] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3960.494723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3965.924138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3965.925952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.927976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3965.945145] Lustre: Skipped 1 previous similar message [ 3971.043880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.051633] Lustre: Skipped 2 previous similar messages [ 3976.163960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.168702] Lustre: Skipped 1 previous similar message [ 3980.255285] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3980.329277] Lustre: server umount lustre-OST0000 complete [ 3986.401293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.402851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.406701] Lustre: Skipped 1 previous similar message [ 3986.411192] Lustre: Skipped 1 previous similar message [ 3991.520728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3991.527958] Lustre: Skipped 1 previous similar message [ 3996.127154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3996.199132] Lustre: server umount lustre-MDT0000 complete [ 3996.641531] LustreError: 277987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.651629] LustreError: 277987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3997.706482] LustreError: 277949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357087 with bad export cookie 2914761552407205792 [ 3997.709665] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.712784] LustreError: 277949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3997.850188] Lustre: server umount lustre-MDT0001 complete [ 4002.012138] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4003.154652] Key type lgssc unregistered [ 4003.297457] LNet: 281681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.301397] LNetError: 281681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.309354] LNet: Removed LNI 192.168.203.142@tcp [ 4003.662984] Key type .llcrypt unregistered [ 4003.664592] Key type ._llcrypt unregistered [ 4008.367438] Key type ._llcrypt registered [ 4008.369477] Key type .llcrypt registered [ 4008.421748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4015.342983] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4022.198250] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4022.600606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.622852] alg: No test for adler32 (adler32-zlib) [ 4023.491633] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4023.593544] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4025.183160] Key type lgssc registered [ 4025.653639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4039.671673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4039.686212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.804781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4040.820593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4040.862147] Lustre: lustre-MDT0000: new disk, initializing [ 4040.889925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.898536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4042.462706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4046.984568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.032431] Lustre: 285208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4047.072460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4047.074880] Lustre: Skipped 1 previous similar message [ 4047.114622] Lustre: lustre-MDT0001: new disk, initializing [ 4047.154522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.170245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4047.174758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4048.695734] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4052.667155] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4054.673054] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4057.205254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4057.308779] Lustre: lustre-OST0000: new disk, initializing [ 4057.310783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4057.314994] Lustre: 287179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4057.350807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.501516] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4063.595553] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4064.253100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4064.257536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4064.298762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4065.341227] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4065.430916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4067.116585] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4067.222575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4069.344824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.345108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.347472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.348019] LustreError: Skipped 1 previous similar message [ 4069.353361] Lustre: Skipped 1 previous similar message [ 4070.045361] Lustre: server umount lustre-OST0000 complete [ 4072.929136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.933177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.939713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.943228] Lustre: Skipped 1 previous similar message [ 4074.464147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.464823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.667148] Lustre: server umount lustre-MDT0000 complete [ 4079.114436] LustreError: 286004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357169 with bad export cookie 1676913923454284777 [ 4079.117479] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.119698] LustreError: 286004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4079.249312] Lustre: server umount lustre-MDT0001 complete [ 4082.652934] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:52:52 (1788357172) [ 4085.183124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.376669] LustreError: 289588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.385893] LustreError: 289588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4085.411669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.912683] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4090.129383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.289464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.929789] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4094.711089] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4096.809717] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4099.364354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.493371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.727607] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4104.617945] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4121.059490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.059506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4121.060884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.060890] Lustre: Skipped 1 previous similar message [ 4121.066228] Lustre: Skipped 1 previous similar message [ 4124.625266] Lustre: server umount lustre-OST0000 complete [ 4126.176384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4126.176990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.179613] LustreError: Skipped 1 previous similar message [ 4126.180302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4126.184882] Lustre: Skipped 1 previous similar message [ 4126.186846] Lustre: Skipped 1 previous similar message [ 4132.195838] Lustre: server umount lustre-MDT0000 complete [ 4133.818704] LustreError: 289570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357223 with bad export cookie 1676913923454285652 [ 4133.821602] LustreError: 290327:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4133.822189] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.824833] LustreError: 289570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.969543] Lustre: server umount lustre-MDT0001 complete [ 4138.127160] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4139.368962] Key type lgssc unregistered [ 4139.506445] LNet: 293140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4139.510397] LNetError: 293140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4139.521532] LNet: Removed LNI 192.168.203.142@tcp [ 4139.865881] Key type .llcrypt unregistered [ 4139.867477] Key type ._llcrypt unregistered [ 4150.289370] Key type ._llcrypt registered [ 4150.290978] Key type .llcrypt registered [ 4150.338331] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4157.327153] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4157.713665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4157.775525] alg: No test for adler32 (adler32-zlib) [ 4158.663085] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4158.772432] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4160.367309] Key type lgssc registered [ 4160.824026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4178.887913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4178.899339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.013420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4180.027384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4180.066460] Lustre: lustre-MDT0000: new disk, initializing [ 4180.101810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.111059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4181.745085] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4186.406425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.453976] Lustre: 296937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4186.481383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4186.483502] Lustre: Skipped 1 previous similar message [ 4186.521513] Lustre: lustre-MDT0001: new disk, initializing [ 4186.565485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4186.579680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4186.583301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4188.062049] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4192.412966] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4194.605269] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4197.248134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.356809] Lustre: lustre-OST2710: new disk, initializing [ 4197.359804] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4197.363566] Lustre: 298903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4197.394795] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4199.408104] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4199.995330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4199.999845] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4200.044508] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4203.410208] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4205.876858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.944904] Lustre: lustre-OST03e8: new disk, initializing [ 4205.947759] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4205.951069] Lustre: 299968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.982228] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4207.873810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4207.878181] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4207.917736] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4208.081841] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4210.838201] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4214.950138] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4215.064978] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.834459] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4216.921382] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4218.518186] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4218.597835] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4220.254797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4220.351446] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.431798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4222.435562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.441823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.459912] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.460445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.469916] Lustre: Skipped 3 previous similar messages [ 4227.932943] Lustre: server umount lustre-MDT0000 complete [ 4229.269153] LustreError: 296930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357319 with bad export cookie 2136075536561801916 [ 4229.272495] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4229.274150] LustreError: 296930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4229.395084] Lustre: server umount lustre-MDT0001 complete [ 4240.906480] Lustre: server umount lustre-OST2710 complete [ 4252.507785] Lustre: server umount lustre-OST03e8 complete [ 4258.438466] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4261.625079] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4273.680619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.762600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4273.773807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4273.808260] Lustre: lustre-MDT0000: new disk, initializing [ 4273.829723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.837280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4275.223838] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4279.692535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.742704] Lustre: 305206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4279.852118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4281.451541] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4285.667929] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4287.752588] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4290.384583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.495988] Lustre: lustre-OST0000: new disk, initializing [ 4290.498268] Lustre: Skipped 1 previous similar message [ 4290.500645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4290.504786] Lustre: Skipped 2 previous similar messages [ 4290.507770] Lustre: 307174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4290.539127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.543722] Lustre: Skipped 1 previous similar message [ 4292.670179] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4296.552152] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4298.116888] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4299.778267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4299.782253] Lustre: Skipped 1 previous similar message [ 4299.784986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4299.818313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4300.260953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4301.942706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4302.047876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4304.865335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4304.868714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.870589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.874656] Lustre: Skipped 3 previous similar messages [ 4309.014833] Lustre: server umount lustre-OST0000 complete [ 4315.106767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.107614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4315.111635] Lustre: Skipped 1 previous similar message [ 4315.114227] Lustre: Skipped 1 previous similar message [ 4316.841626] Lustre: server umount lustre-MDT0000 complete [ 4318.547275] LustreError: 305199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357408 with bad export cookie 2136075536561803596 [ 4318.549575] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4318.556847] LustreError: 305199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4318.715676] Lustre: server umount lustre-MDT0001 complete [ 4322.614937] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:56:52 (1788357412) [ 4323.257491] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4324.041584] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:56:54 (1788357414) [ 4325.399260] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4337.703114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.845257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4337.860100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.967758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.907654] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4344.244932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.312053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4344.330368] Lustre: 311037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4346.338823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4363.242848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4363.339266] Lustre: Denying initial registration attempt from nid 192.168.203.142@tcp, specified as failover [ 4363.344516] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4363.349272] LustreError: 312484:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4363.352648] LustreError: 312484:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4363.356055] LustreError: 312484:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4363.387326] Lustre: server umount lustre-OST0000 complete [ 4363.389085] LustreError: 312484:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4366.816243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.818972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.822807] Lustre: Skipped 1 previous similar message [ 4366.829089] Lustre: Skipped 1 previous similar message [ 4372.509052] Lustre: server umount lustre-MDT0000 complete [ 4374.027189] LustreError: 311029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357464 with bad export cookie 2136075536561804450 [ 4374.029836] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.032157] LustreError: 311029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4374.150362] Lustre: server umount lustre-MDT0001 complete [ 4378.378168] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4379.694279] Key type lgssc unregistered [ 4379.862535] LNet: 313610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4379.865819] LNetError: 313610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4379.876318] LNet: Removed LNI 192.168.203.142@tcp [ 4380.304335] Key type .llcrypt unregistered [ 4380.307119] Key type ._llcrypt unregistered [ 4384.998810] Key type ._llcrypt registered [ 4385.000134] Key type .llcrypt registered [ 4385.045162] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4385.475494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4385.507534] alg: No test for adler32 (adler32-zlib) [ 4386.398238] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4386.503204] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4388.103260] Key type lgssc registered [ 4388.606591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4415.066473] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4418.642297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4418.649367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.742834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4419.751723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4419.810361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.362798] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4424.607681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.645086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4424.659743] Lustre: 316348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4424.761526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.212152] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4428.689748] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4430.577352] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4445.922297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.997193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4446.074534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.171832] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4450.646736] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4451.447689] Lustre: server umount lustre-OST0000 complete [ 4452.853249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.859117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.861782] Lustre: Skipped 1 previous similar message [ 4457.952034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.954408] Lustre: Skipped 1 previous similar message [ 4462.048287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.052193] Lustre: Skipped 1 previous similar message [ 4467.167145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4467.168831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.178126] Lustre: Skipped 1 previous similar message [ 4467.241875] Lustre: server umount lustre-MDT0000 complete [ 4468.780302] LustreError: 316340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357558 with bad export cookie 1555617086934148741 [ 4468.783603] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.785478] LustreError: 316340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4468.912748] Lustre: server umount lustre-MDT0001 complete [ 4473.097470] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4474.250858] Key type lgssc unregistered [ 4474.416374] LNet: 319911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4474.420212] LNetError: 319911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4474.433347] LNet: Removed LNI 192.168.203.142@tcp [ 4474.764696] Key type .llcrypt unregistered [ 4474.766201] Key type ._llcrypt unregistered [ 4484.999762] Key type ._llcrypt registered [ 4485.001504] Key type .llcrypt registered [ 4485.048448] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4485.482908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4485.504463] alg: No test for adler32 (adler32-zlib) [ 4486.378130] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4486.489289] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4488.081895] Key type lgssc registered [ 4488.558697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4491.690168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4491.696614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.862790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.404303] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4497.654514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.820767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4499.429267] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4502.235458] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.567905] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.347453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.493492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.990310] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4514.972645] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4531.677916] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 4533.217177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.218898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4533.222509] Lustre: Skipped 2 previous similar messages [ 4533.227071] Lustre: Skipped 2 previous similar messages [ 4535.204448] Lustre: server umount lustre-MDT0000 complete [ 4536.628057] LustreError: 323146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357626 with bad export cookie 5682077254852942039 [ 4536.631273] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.632713] LustreError: 323146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4536.788838] LustreError: 322032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.796232] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 4536.796488] LustreError: 322032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4538.335990] LustreError: 321299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.336350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.344567] LustreError: 323752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.344702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4541.355015] Lustre: Skipped 1 previous similar message [ 4542.834136] Lustre: server umount lustre-MDT0001 complete [ 4545.353395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4548.698647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.888323] LustreError: 320616:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000052090219 x1875228833636864/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4562.028614] LustreError: 323133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.040459] LustreError: 323133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4562.064279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.095931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4563.708680] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4566.177677] LustreError: 323134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.183676] LustreError: 323134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4567.002948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.162376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4568.756622] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4571.236597] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4571.505462] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4573.093744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4575.095073] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4575.666554] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4577.760389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4577.760475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.764304] LustreError: Skipped 1 previous similar message [ 4577.771215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4577.776518] Lustre: Skipped 2 previous similar messages [ 4583.267910] Lustre: server umount lustre-OST0000 complete [ 4586.976446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.977283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.982283] Lustre: Skipped 2 previous similar messages [ 4586.984283] Lustre: Skipped 2 previous similar messages [ 4590.814913] Lustre: server umount lustre-MDT0000 complete [ 4592.096148] LustreError: 324919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.102721] LustreError: 324919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4592.172554] LustreError: 321286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357682 with bad export cookie 5682077254852949193 [ 4592.176193] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.177828] LustreError: 321286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4592.295407] Lustre: server umount lustre-MDT0001 complete [ 4596.159735] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4597.322927] Key type lgssc unregistered [ 4597.469541] LNet: 327597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4597.473659] LNetError: 327597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4597.485658] LNet: Removed LNI 192.168.203.142@tcp [ 4597.852288] Key type .llcrypt unregistered [ 4597.853852] Key type ._llcrypt unregistered [ 4607.999093] Key type ._llcrypt registered [ 4608.000233] Key type .llcrypt registered [ 4608.040269] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4608.479406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4608.488303] alg: No test for adler32 (adler32-zlib) [ 4609.350901] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4609.438514] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4611.015113] Key type lgssc registered [ 4611.376619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4614.026251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4614.033444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.191141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.944567] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4620.604175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.762632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.369756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4625.268388] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4627.292881] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4630.624691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.629725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4630.632349] Lustre: Skipped 1 previous similar message [ 4634.721301] Lustre: server umount lustre-MDT0000 complete [ 4636.129555] LustreError: 328985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.136029] LustreError: 328985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4636.208271] LustreError: 328970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357726 with bad export cookie 6526492467002022069 [ 4636.211702] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.213238] LustreError: 328970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.332029] Lustre: server umount lustre-MDT0001 complete [ 4641.961212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.074249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4642.091825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4642.168775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.665102] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4647.020417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.073069] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4647.085488] Lustre: 331578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4647.190158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.746026] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4651.730486] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4654.017649] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4659.301298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.398859] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4659.405334] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4659.410760] LustreError: 333461:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4659.413774] LustreError: 333461:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4659.416506] LustreError: 333461:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4659.443841] Lustre: server umount lustre-OST0000 complete [ 4659.445771] LustreError: 333461:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4665.058968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4665.157359] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4665.246848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.508600] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4670.238891] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4672.533311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4672.884838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4672.915566] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4672.943728] Lustre: lustre-OST0001: new disk, initializing [ 4672.945271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4672.948199] Lustre: 334999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4674.906885] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4677.414649] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4678.149804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4678.154443] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4678.186398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4683.234978] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4683.238170] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.243152] Lustre: Skipped 1 previous similar message [ 4683.246284] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4683.248374] Lustre: Skipped 1 previous similar message [ 4684.541034] Lustre: server umount lustre-OST0001 complete [ 4687.840159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.840184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.843832] LustreError: Skipped 2 previous similar messages [ 4687.848429] Lustre: Skipped 1 previous similar message [ 4687.850859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.853966] Lustre: Skipped 1 previous similar message [ 4688.354717] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.359578] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4693.474060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4693.475166] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.476538] Lustre: Skipped 3 previous similar messages [ 4693.486433] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4698.597236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4698.597931] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.600300] Lustre: Skipped 1 previous similar message [ 4698.608942] LustreError: 333927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4700.127135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4700.182570] Lustre: server umount lustre-OST0000 complete [ 4703.713017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4703.714417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.720000] Lustre: Skipped 1 previous similar message [ 4707.614565] Lustre: server umount lustre-MDT0000 complete [ 4708.832292] LustreError: 331584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.838311] LustreError: 331584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4708.890217] LustreError: 331569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357799 with bad export cookie 6526492467002022615 [ 4708.893050] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.895310] LustreError: 331569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4709.011623] Lustre: server umount lustre-MDT0001 complete [ 4712.545457] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4713.609775] Key type lgssc unregistered [ 4713.739858] LNet: 336959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4713.743110] LNetError: 336959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4713.752486] LNet: Removed LNI 192.168.203.142@tcp [ 4714.049547] Key type .llcrypt unregistered [ 4714.050832] Key type ._llcrypt unregistered [ 4747.168161] Key type ._llcrypt registered [ 4747.169212] Key type .llcrypt registered [ 4747.207233] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4753.631780] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4754.070690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4754.084967] alg: No test for adler32 (adler32-zlib) [ 4754.950650] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4755.045652] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4756.631167] Key type lgssc registered [ 4757.009142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4768.644084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4768.651055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.734774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4769.747081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4769.784249] Lustre: lustre-MDT0000: new disk, initializing [ 4769.807532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.815633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4771.200455] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4775.660165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.703072] Lustre: 341858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4775.730569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4775.733299] Lustre: Skipped 1 previous similar message [ 4775.774858] Lustre: lustre-MDT0001: new disk, initializing [ 4775.809370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4775.823188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4775.827726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4777.316689] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4781.269553] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4783.278405] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4785.805454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.913444] Lustre: lustre-OST0000: new disk, initializing [ 4785.915401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4785.920272] Lustre: 343830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4785.952761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.642353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4787.646285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4787.677884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.895880] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4791.512539] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4792.880937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4792.958895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4794.373738] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4794.445438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4797.210669] Lustre: server umount lustre-OST0000 complete [ 4801.503816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4801.506156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.511974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.040222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.040770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.160366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.162879] Lustre: Skipped 2 previous similar messages [ 4808.191141] Lustre: server umount lustre-MDT0000 complete [ 4809.595558] LustreError: 341852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788357899 with bad export cookie 10638470077237327440 [ 4809.598456] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4809.599894] LustreError: 341852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4818.966712] Lustre: server umount lustre-MDT0001 complete [ 4822.009803] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:05:12 (1788357912) [ 4824.631794] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:05:14 (1788357914) [ 4827.769334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.961733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.333832] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4832.443630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.572404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4833.946793] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4836.446270] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.408168] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4840.726269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4840.859461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.899508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4845.483943] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4859.180488] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4859.702720] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4860.253539] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4860.822945] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4861.499847] Lustre: Failing over lustre-MDT0000 [ 4861.667943] Lustre: server umount lustre-MDT0000 complete [ 4863.455854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4863.458670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.463040] LustreError: 346579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.469164] LustreError: 346579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4864.480232] LustreError: 347309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.487470] LustreError: 347309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4866.011570] LustreError: 346575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.600644] LustreError: 346580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.606592] LustreError: 346580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4874.720442] LustreError: 346579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4874.726267] LustreError: 346579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4874.729218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.772618] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4874.862672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.890198] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4876.240809] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4876.244433] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4880.361740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4880.369390] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4880.384310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4882.207513] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4882.731753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4884.544994] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4885.192410] Lustre: Failing over lustre-MDT0000 [ 4885.362627] Lustre: server umount lustre-MDT0000 complete [ 4885.472231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.472603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4885.473942] LustreError: 349067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.475638] Lustre: Skipped 3 previous similar messages [ 4898.340579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.390563] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.531171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4899.996340] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4901.844948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4903.909801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4903.911643] Lustre: Skipped 2 previous similar messages [ 4903.920847] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4903.936554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4905.663753] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4906.145984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4909.025449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4909.025620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4909.026424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4909.029033] Lustre: Skipped 2 previous similar messages [ 4914.144095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4914.148850] Lustre: Skipped 4 previous similar messages [ 4914.454746] Lustre: server umount lustre-MDT0000 complete [ 4915.650053] LustreError: 350561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358005 with bad export cookie 10638470077237330121 [ 4915.652571] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.653011] LustreError: 350561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4915.767852] Lustre: server umount lustre-MDT0001 complete [ 4927.197367] Lustre: server umount lustre-OST0000 complete [ 4932.309585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.455934] LustreError: 352928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.463043] LustreError: 352928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4932.478082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.480551] Lustre: Skipped 1 previous similar message [ 4933.737717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4936.421644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.800221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4940.055872] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4941.734666] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4943.787544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4945.738554] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4948.004954] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4957.220715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4961.247865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4961.251666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.258168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4965.968330] Lustre: server umount lustre-OST0000 complete [ 4973.267450] Lustre: server umount lustre-MDT0000 complete [ 4974.470066] LustreError: 352910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358064 with bad export cookie 10638470077237330842 [ 4974.472739] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.474292] LustreError: 352910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4974.581928] Lustre: server umount lustre-MDT0001 complete [ 4977.947144] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4978.939406] Key type lgssc unregistered [ 4979.071396] LNet: 356478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4979.073905] LNetError: 356478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4979.082305] LNet: Removed LNI 192.168.203.142@tcp [ 4979.384129] Key type .llcrypt unregistered [ 4979.385374] Key type ._llcrypt unregistered [ 4988.406234] Key type ._llcrypt registered [ 4988.407666] Key type .llcrypt registered [ 4988.442909] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4988.728604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4988.754240] alg: No test for adler32 (adler32-zlib) [ 4989.599120] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 4989.675551] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4991.255164] Key type lgssc registered [ 4991.582375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.011678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4994.017432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.150144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.394826] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4999.023302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.130108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5000.395873] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5002.589645] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5004.269745] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5006.352531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.454438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.208514] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5010.344564] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5020.771283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5029.855702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5029.858510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.862268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5031.393143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5031.393228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.395253] Lustre: Skipped 1 previous similar message [ 5031.399503] Lustre: Skipped 1 previous similar message [ 5035.539559] Lustre: server umount lustre-MDT0000 complete [ 5036.512193] LustreError: 360460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.518394] LustreError: 360460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5036.721780] LustreError: 357955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358126 with bad export cookie 17961422429697509131 [ 5036.725410] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.726189] LustreError: 357955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.850845] Lustre: server umount lustre-MDT0001 complete [ 5048.345576] Lustre: server umount lustre-OST0000 complete [ 5052.796414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.957261] LustreError: 361702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.983574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.410216] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5057.262208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.370827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5058.671936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5060.954594] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5062.752267] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5064.918277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5066.848854] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5069.198073] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5078.379160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5088.224198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.227565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.232169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.234189] Lustre: Skipped 1 previous similar message [ 5092.500308] Lustre: server umount lustre-MDT0000 complete [ 5093.746437] LustreError: 361684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358183 with bad export cookie 17961422429697510237 [ 5093.749426] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.752261] LustreError: 361684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5093.869316] Lustre: server umount lustre-MDT0001 complete [ 5105.371989] Lustre: server umount lustre-OST0000 complete [ 5110.530907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.687878] LustreError: 365627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.711520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.714190] Lustre: Skipped 1 previous similar message [ 5112.077747] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5114.869582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.414935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5118.890520] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.810300] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5123.060136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5125.211421] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5127.762685] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5135.461660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5140.961828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5140.963163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.963804] LustreError: Skipped 1 previous similar message [ 5140.969764] Lustre: Skipped 2 previous similar messages [ 5140.971857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5140.973707] Lustre: Skipped 3 previous similar messages [ 5141.715028] Lustre: server umount lustre-OST0000 complete [ 5145.172664] Lustre: server umount lustre-MDT0000 complete [ 5146.081301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.081897] LustreError: 365628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.085531] Lustre: Skipped 1 previous similar message [ 5146.094734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5146.441215] LustreError: 365609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358236 with bad export cookie 17961422429697511238 [ 5146.445343] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.447021] LustreError: 365609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5146.567773] Lustre: server umount lustre-MDT0001 complete [ 5149.915266] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5150.926776] Key type lgssc unregistered [ 5151.050336] LNet: 369179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5151.052661] LNetError: 369179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5151.060348] LNet: Removed LNI 192.168.203.142@tcp [ 5151.407856] Key type .llcrypt unregistered [ 5151.409065] Key type ._llcrypt unregistered [ 5161.083689] Key type ._llcrypt registered [ 5161.085203] Key type .llcrypt registered [ 5161.123620] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5161.594543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5161.604800] alg: No test for adler32 (adler32-zlib) [ 5162.465445] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5162.571267] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5164.167157] Key type lgssc registered [ 5164.552348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5167.178554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5167.185522] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.189076] LustreError: 370597:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5167.198175] LustreError: 370597:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5167.202406] LustreError: 370597:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5167.206086] LustreError: 370597:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5167.211293] LustreError: 370597:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5169.753758] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5169.756548] LustreError: 370993:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5169.766193] LustreError: 370993:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5169.768812] LustreError: 370993:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5169.771702] LustreError: 370993:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5169.775125] LustreError: 370993:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5175.950870] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5176.965195] Key type lgssc unregistered [ 5177.094353] LNet: 371994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5177.097356] LNetError: 371994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5177.110353] LNet: Removed LNI 192.168.203.142@tcp [ 5177.419914] Key type .llcrypt unregistered [ 5177.421622] Key type ._llcrypt unregistered [ 5183.306309] Key type ._llcrypt registered [ 5183.308272] Key type .llcrypt registered [ 5183.346353] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5189.388939] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5189.706320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5189.739082] alg: No test for adler32 (adler32-zlib) [ 5190.588607] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5190.676638] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5192.255155] Key type lgssc registered [ 5192.600914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5203.834905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5203.843110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.926460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5204.936977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5204.977447] Lustre: lustre-MDT0000: new disk, initializing [ 5205.000032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.008559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5206.385184] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5210.599952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.641981] Lustre: 375122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5210.667797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5210.669414] Lustre: Skipped 1 previous similar message [ 5210.710237] Lustre: lustre-MDT0001: new disk, initializing [ 5210.744479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5210.758129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5210.762920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5212.212258] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5215.964796] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5217.753270] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5220.062474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.166924] Lustre: lustre-OST0000: new disk, initializing [ 5220.169545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5220.173973] Lustre: 377091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5220.208773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.173835] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5226.065750] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5226.493401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5226.497072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5226.529603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5227.495274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5227.582501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5228.983306] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5229.062330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5231.587634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.587960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.590275] LustreError: Skipped 1 previous similar message [ 5231.590919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5231.596404] Lustre: Skipped 1 previous similar message [ 5235.861867] Lustre: server umount lustre-OST0000 complete [ 5241.824956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5241.825025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.825731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5241.825736] Lustre: Skipped 1 previous similar message [ 5241.836603] Lustre: Skipped 1 previous similar message [ 5243.355606] Lustre: server umount lustre-MDT0000 complete [ 5244.662546] LustreError: 375117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358334 with bad export cookie 2099204867797365417 [ 5244.665850] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.666582] LustreError: 375117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5244.793683] Lustre: server umount lustre-MDT0001 complete [ 5247.635785] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5248.121308] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:12:18 (1788358338) [ 5250.247596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.423690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.672364] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5254.281337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.511171] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5257.542372] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5259.095147] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5261.091938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.183693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.185613] Lustre: Skipped 1 previous similar message [ 5262.901909] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5265.060627] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5267.133103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.189930] Lustre: lustre-OST0001: new disk, initializing [ 5267.191448] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5267.194976] Lustre: 382380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5268.219296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5268.222841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5268.250734] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5268.974958] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5272.567807] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.425638] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.428553] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5275.430733] Lustre: Skipped 1 previous similar message [ 5275.433802] Lustre: Skipped 1 previous similar message [ 5279.199629] Lustre: lustre-OST0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 5279.202742] Lustre: Skipped 1 previous similar message [ 5280.451080] Lustre: server umount lustre-OST0001 complete [ 5280.739192] LustreError: 381368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.744689] LustreError: 381368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5285.857577] LustreError: 381370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.865313] LustreError: 381370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5290.977677] LustreError: 381368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.993090] LustreError: 381368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5295.072638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5295.076731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.084994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.100448] LustreError: 382567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.122138] LustreError: 382567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5301.224159] LustreError: 381946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.242353] LustreError: 381946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5306.340599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5306.343529] Lustre: Skipped 5 previous similar messages [ 5307.359224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5307.498803] Lustre: server umount lustre-OST0000 complete [ 5310.945468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5310.953540] LustreError: Skipped 1 previous similar message [ 5310.956431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.965155] Lustre: Skipped 1 previous similar message [ 5316.659859] Lustre: server umount lustre-MDT0000 complete [ 5318.208468] LustreError: 379529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358408 with bad export cookie 2099204867797366285 [ 5318.212771] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5318.214817] LustreError: 379529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5324.423356] Lustre: server umount lustre-MDT0001 complete [ 5328.466431] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5329.671485] Key type lgssc unregistered [ 5329.818427] LNet: 384364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5329.821721] LNetError: 384364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5329.829086] LNet: Removed LNI 192.168.203.142@tcp [ 5330.201045] Key type .llcrypt unregistered [ 5330.202481] Key type ._llcrypt unregistered [ 5350.846419] Key type ._llcrypt registered [ 5350.847856] Key type .llcrypt registered [ 5350.896608] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5351.429838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5351.439129] alg: No test for adler32 (adler32-zlib) [ 5352.311987] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5352.429978] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5354.023137] Key type lgssc registered [ 5354.426899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5357.453654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5357.462718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.575474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5358.586058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5358.653295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.243499] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5363.472996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.519552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5363.533535] Lustre: 387139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5363.646357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5365.334751] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5368.256673] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5370.397909] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5371.872406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.873187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.876915] Lustre: Skipped 1 previous similar message [ 5376.992106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5376.995652] Lustre: Skipped 2 previous similar messages [ 5377.245097] Lustre: server umount lustre-MDT0000 complete [ 5378.652157] LustreError: 387131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358468 with bad export cookie 2049799895402114159 [ 5378.655038] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5378.656807] LustreError: 387131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5378.784540] Lustre: server umount lustre-MDT0001 complete [ 5382.254215] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:14:32 (1788358472) [ 5384.654476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.825331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.157285] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5389.340578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.543734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5391.051893] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5393.650417] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.717526] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5398.441213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5398.543148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5398.612919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.661138] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5403.283506] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5407.207518] Lustre: Permanently deactivating lustre-OST0000 [ 5407.214183] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5416.463820] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5417.825397] LustreError: 392775:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5417.832028] LustreError: 392775:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5417.835879] LustreError: 392775:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5420.523674] LustreError: 392823:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5420.528584] LustreError: 392823:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5425.626296] Lustre: server umount lustre-OST0000 complete [ 5427.912729] LustreError: 393072:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5427.917414] LustreError: 393072:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5430.752549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5430.758490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.767687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5431.780893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.899831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.950132] Lustre: Skipped 3 previous similar messages [ 5439.142444] Lustre: server umount lustre-MDT0000 complete [ 5442.020349] LustreError: 389648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.037787] LustreError: 389648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5443.061121] LustreError: 391478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358533 with bad export cookie 2049799895402114726 [ 5443.070502] LustreError: 391478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5443.076308] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.880123] Lustre: server umount lustre-MDT0001 complete [ 5463.956275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.641380] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5471.320953] LustreError: 394230:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5476.471717] Lustre: 394569:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.142@tcp [ 5477.911643] Lustre: server umount MGS complete [ 5485.516167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.854929] LustreError: 395293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.874071] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5485.881093] Lustre: Skipped 1 previous similar message [ 5485.955554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.516354] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5491.172769] LustreError: 395294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.992435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.198699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5498.638523] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5503.782261] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5506.582300] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5507.393974] Lustre: Permanently reactivating lustre-OST0000 [ 5507.400778] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5507.403579] Lustre: Skipped 2 previous similar messages [ 5535.199165] LustreError: 397171:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5540.195736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5540.398680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.403920] Lustre: Skipped 1 previous similar message [ 5543.710140] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5547.865582] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5555.680788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5555.682025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.687695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5555.695833] Lustre: Skipped 1 previous similar message [ 5555.704782] Lustre: Skipped 1 previous similar message [ 5557.672259] Lustre: server umount lustre-OST0000 complete [ 5560.803793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.812403] Lustre: Skipped 1 previous similar message [ 5561.433358] Lustre: server umount lustre-MDT0000 complete [ 5563.165877] LustreError: 395274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358653 with bad export cookie 2049799895402115762 [ 5563.168084] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.171578] LustreError: 395274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5563.335911] Lustre: server umount lustre-MDT0001 complete [ 5567.938910] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5569.213909] Key type lgssc unregistered [ 5569.362437] LNet: 399944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5569.365945] LNetError: 399944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5569.373753] LNet: Removed LNI 192.168.203.142@tcp [ 5569.749201] Key type .llcrypt unregistered [ 5569.751035] Key type ._llcrypt unregistered [ 5577.096494] Key type ._llcrypt registered [ 5577.098404] Key type .llcrypt registered [ 5577.150909] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5584.886391] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5585.293953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5585.323688] alg: No test for adler32 (adler32-zlib) [ 5586.228985] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5586.340060] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5587.927208] Key type lgssc registered [ 5588.428402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5604.229954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5604.243216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.348780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5605.366678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5605.432700] Lustre: lustre-MDT0000: new disk, initializing [ 5605.488783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.498423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5607.286665] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5612.767417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.842566] Lustre: 403090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5612.880750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5612.885861] Lustre: Skipped 1 previous similar message [ 5612.926443] Lustre: lustre-MDT0001: new disk, initializing [ 5612.964940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.984786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5612.991157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5614.826355] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5619.635155] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5622.136952] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5625.485506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5625.623862] Lustre: lustre-OST0000: new disk, initializing [ 5625.628254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5625.633481] Lustre: 405054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5625.692155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.442206] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5633.404475] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5633.544745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5633.549911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5633.615428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5635.598674] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5635.722605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5637.635352] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5637.740427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5643.744515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5643.745435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.748497] LustreError: Skipped 1 previous similar message [ 5643.750827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5643.756174] Lustre: Skipped 1 previous similar message [ 5643.765807] Lustre: Skipped 1 previous similar message [ 5644.857799] Lustre: server umount lustre-OST0000 complete [ 5648.864786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.866294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.866579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.873836] Lustre: Skipped 1 previous similar message [ 5653.985277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.988817] Lustre: Skipped 1 previous similar message [ 5659.104716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5659.111164] Lustre: Skipped 1 previous similar message [ 5661.151339] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5661.255931] Lustre: server umount lustre-MDT0000 complete [ 5663.458112] LustreError: 403083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358753 with bad export cookie 3730590421646319558 [ 5663.460777] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.466747] LustreError: 403083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5663.662856] Lustre: server umount lustre-MDT0001 complete [ 5668.591371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5669.623951] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:19:19 (1788358759) [ 5673.676219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.970950] LustreError: 407513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.984845] LustreError: 407513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5674.027986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.279935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5679.073247] LustreError: 407513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.087349] LustreError: 407513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5680.716272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.955228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5683.071128] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5687.799759] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5691.165859] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5694.877766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.061970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.295116] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5702.673156] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5703.731945] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5705.546608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5714.912966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.918175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.929434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.962641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.966362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.249452] Lustre: server umount lustre-OST0000 complete [ 5722.081553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.081915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5722.085938] Lustre: Skipped 1 previous similar message [ 5727.200199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.207293] Lustre: Skipped 5 previous similar messages [ 5735.391150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5735.473945] Lustre: server umount lustre-MDT0000 complete [ 5737.274773] LustreError: 410282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358827 with bad export cookie 3730590421646320426 [ 5737.275137] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.282341] LustreError: 410282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5737.426761] Lustre: server umount lustre-MDT0001 complete [ 5742.631332] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5744.025943] Key type lgssc unregistered [ 5744.200623] LNet: 411237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5744.211069] LNetError: 411237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5744.225838] LNet: Removed LNI 192.168.203.142@tcp [ 5744.664224] Key type .llcrypt unregistered [ 5744.666143] Key type ._llcrypt unregistered [ 5756.457970] Key type ._llcrypt registered [ 5756.460204] Key type .llcrypt registered [ 5756.522176] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5756.678347] Key type .llcrypt unregistered [ 5756.681188] Key type ._llcrypt unregistered [ 5766.188496] Key type ._llcrypt registered [ 5766.190690] Key type .llcrypt registered [ 5766.256620] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5766.828149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.839912] alg: No test for adler32 (adler32-zlib) [ 5767.743196] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5767.886807] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5769.503171] Key type lgssc registered [ 5770.167148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5774.349437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5774.361307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.609248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5777.864851] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5782.446497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.660265] LustreError: 413428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.671458] LustreError: 413428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5782.682910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.765355] LustreError: 413427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.778500] LustreError: 413427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5785.780539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5785.860820] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5789.839438] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5791.203240] LustreError: 414170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.220095] LustreError: 414170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5793.611766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.822349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.107255] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5806.560302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5806.563809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.572026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.122943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.124091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.128423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.006626] Lustre: server umount lustre-OST0000 complete [ 5814.242182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.243911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5814.249243] Lustre: Skipped 1 previous similar message [ 5814.251571] Lustre: Skipped 1 previous similar message [ 5814.258741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5815.319431] Lustre: server umount lustre-MDT0000 complete [ 5817.491128] LustreError: 413409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358907 with bad export cookie 18313252332670494827 [ 5817.493492] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.500587] LustreError: 413409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5817.666934] Lustre: server umount lustre-MDT0001 complete [ 5823.925063] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5825.829941] Key type lgssc unregistered [ 5826.029443] LNet: 416573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5826.032717] LNetError: 416573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5826.043437] LNet: Removed LNI 192.168.203.142@tcp [ 5826.564201] Key type .llcrypt unregistered [ 5826.569674] Key type ._llcrypt unregistered [ 5841.727579] Key type ._llcrypt registered [ 5841.729842] Key type .llcrypt registered [ 5841.803714] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5842.357967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5842.416503] alg: No test for adler32 (adler32-zlib) [ 5843.359371] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 5843.497991] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5845.119180] Key type lgssc registered [ 5845.791672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5850.167745] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5850.189565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5865.696815] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5870.815542] LustreError: 417941:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 5897.439642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.937435] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5914.273945] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5923.375935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.781335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.852627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5928.224823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5928.941854] LustreError: 417967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.994965] LustreError: 417967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5932.029596] LustreError: 418983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.087752] LustreError: 418983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5937.124857] LustreError: 417966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.141917] LustreError: 417966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5941.292472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.869893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5953.739442] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5964.022631] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5973.899414] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5985.250706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.256057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.267620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.328975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.338208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.345833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.850240] Lustre: server umount lustre-OST0000 complete [ 5997.135932] Lustre: server umount lustre-MDT0000 complete [ 5998.049869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.073991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.115201] LustreError: 418987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.556049] LustreError: 417948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359091 with bad export cookie 7607040335383011842 [ 6001.558128] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.569347] LustreError: 417948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6001.868867] Lustre: server umount lustre-MDT0001 complete [ 6013.176855] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6016.526847] Key type lgssc unregistered [ 6016.861759] LNet: 421517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6016.870909] LNetError: 421517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6016.900386] LNet: Removed LNI 192.168.203.142@tcp [ 6018.045958] Key type .llcrypt unregistered [ 6018.049233] Key type ._llcrypt unregistered [ 6038.679394] Key type ._llcrypt registered [ 6038.683425] Key type .llcrypt registered [ 6038.771255] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6039.631839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6039.738071] alg: No test for adler32 (adler32-zlib) [ 6040.905949] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 6041.167440] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6042.904270] Key type lgssc registered [ 6044.046760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6050.916300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6050.941383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.421167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.519362] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6062.579136] LustreError: 422914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.600935] LustreError: 422914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6065.134375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.411632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6069.773623] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6077.076072] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.863559] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6089.395337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6089.774242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.604267] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6096.944641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6103.479175] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.232378] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 6111.713435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6111.723310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.746427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6112.228194] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.268990] Lustre: server umount lustre-MDT0000 complete [ 6116.438366] LustreError: 422896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.42@tcp arrived at 1788359206 with bad export cookie 13783037608822913395 [ 6116.450133] LustreError: 422896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6117.350933] LustreError: 422909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.369503] LustreError: 422909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6118.089607] LustreError: 422895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359208 with bad export cookie 13783037608822912499 [ 6118.101308] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6118.103732] LustreError: 422895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6122.464679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.469203] LustreError: 422914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.479084] Lustre: Skipped 1 previous similar message [ 6122.494775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.511913] LustreError: 422914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6122.547209] Lustre: Skipped 3 previous similar messages [ 6124.338957] Lustre: server umount lustre-OST0000 complete [ 6128.612423] LustreError: 422909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.644867] LustreError: 422909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6131.497970] Lustre: server umount lustre-MDT0001 complete [ 6142.005796] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6145.057114] Key type lgssc unregistered [ 6145.366889] LNet: 426597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6145.381510] LNetError: 426597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6145.410662] LNet: Removed LNI 192.168.203.142@tcp [ 6146.320866] Key type .llcrypt unregistered [ 6146.323219] Key type ._llcrypt unregistered [ 6169.605974] Key type ._llcrypt registered [ 6169.609126] Key type .llcrypt registered [ 6169.735829] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6170.766135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6170.999058] alg: No test for adler32 (adler32-zlib) [ 6172.052819] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 6172.286207] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6173.967783] Key type lgssc registered [ 6175.270280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6183.388517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6183.411237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.863506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.401570] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6194.151901] LustreError: 428014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.167324] LustreError: 428014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6197.706228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.104577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.648308] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6209.600574] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6215.531149] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.016386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6222.381483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.572809] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6229.566593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6235.458170] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.340358] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 6244.321425] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6244.331179] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.351946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6244.838113] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.281295] Lustre: server umount lustre-MDT0001 complete [ 6249.953446] LustreError: 428014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.982589] LustreError: 428014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6255.075806] LustreError: 428742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.085092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.118492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.129862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.137170] Lustre: Skipped 1 previous similar message [ 6258.963607] Lustre: server umount lustre-OST0000 complete [ 6260.197771] LustreError: 428015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.058488] Lustre: server umount lustre-MDT0000 complete [ 6276.510201] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6279.166454] Key type lgssc unregistered [ 6279.431670] LNet: 431698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6279.441981] LNetError: 431698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6279.459340] LNet: Removed LNI 192.168.203.142@tcp [ 6280.243397] Key type .llcrypt unregistered [ 6280.247140] Key type ._llcrypt unregistered [ 6301.670401] Key type ._llcrypt registered [ 6301.673868] Key type .llcrypt registered [ 6301.796405] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6313.821727] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6314.575696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.718179] alg: No test for adler32 (adler32-zlib) [ 6315.720918] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 6315.941269] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6317.615482] Key type lgssc registered [ 6318.542583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6346.336868] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6357.808823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6357.824072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.036910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6359.085622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6359.176125] Lustre: lustre-MDT0000: new disk, initializing [ 6359.241929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.260366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6362.639631] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6373.277706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.455510] Lustre: lustre-OST0000: new disk, initializing [ 6373.458918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6373.462671] Lustre: Skipped 1 previous similar message [ 6373.467327] Lustre: 436711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6373.525054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.660704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6377.669182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6377.695157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6378.040155] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6385.270836] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.747291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.846545] Lustre: 435692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6395.893456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6395.960747] Lustre: lustre-MDT0001: new disk, initializing [ 6396.038560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6396.077099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6396.080927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6398.725467] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6405.696071] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6405.848807] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6409.040561] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6409.181250] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6413.290094] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 6416.871718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.880254] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6416.880857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6416.896946] Lustre: Skipped 1 previous similar message [ 6418.391932] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 6418.397637] Lustre: Skipped 1 previous similar message [ 6418.625880] Lustre: server umount lustre-MDT0001 complete [ 6427.106562] LustreError: 435699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6427.119922] LustreError: 435699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6432.227814] LustreError: 435698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.243940] LustreError: 435698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6434.274936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.281167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.290894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.345916] LustreError: 435703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.495667] Lustre: server umount lustre-OST0000 complete [ 6456.287126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6456.473831] Lustre: server umount lustre-MDT0000 complete [ 6465.039591] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6466.894560] Key type lgssc unregistered [ 6467.105754] LNet: 440454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6467.111148] LNetError: 440454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6467.124162] LNet: Removed LNI 192.168.203.142@tcp [ 6467.734495] Key type .llcrypt unregistered [ 6467.737443] Key type ._llcrypt unregistered [ 6494.630997] Key type ._llcrypt registered [ 6494.637022] Key type .llcrypt registered [ 6494.723430] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6494.961388] Key type .llcrypt unregistered [ 6494.963622] Key type ._llcrypt unregistered [ 6505.799145] Key type ._llcrypt registered [ 6505.801110] Key type .llcrypt registered [ 6505.884632] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6506.503216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6506.592175] alg: No test for adler32 (adler32-zlib) [ 6507.583276] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 6507.756107] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6509.415182] Key type lgssc registered [ 6510.326344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6527.224097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6527.284524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.484281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6528.518874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6528.574092] Lustre: lustre-MDT0000: new disk, initializing [ 6528.629730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.643719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6531.360954] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6538.228930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.309138] Lustre: 443809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6538.355361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6538.358271] Lustre: Skipped 1 previous similar message [ 6538.407628] Lustre: lustre-MDT0001: new disk, initializing [ 6538.475576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6538.500767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6538.513728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6541.328345] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6547.790299] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6551.234157] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.445738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.650573] Lustre: lustre-OST0000: new disk, initializing [ 6555.654569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6555.659700] Lustre: 445773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6555.725640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6559.283701] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6565.247847] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6565.398212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6565.405673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6565.473408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6569.441284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6569.445792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.455751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6570.469588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.470050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6570.480488] Lustre: Skipped 1 previous similar message [ 6570.485175] Lustre: Skipped 1 previous similar message [ 6575.515407] Lustre: server umount lustre-MDT0000 complete [ 6577.639813] LustreError: 443800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359667 with bad export cookie 13128585358495033506 [ 6577.640680] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6577.646766] LustreError: 443800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6577.850360] Lustre: server umount lustre-MDT0001 complete [ 6590.257232] Lustre: server umount lustre-OST0000 complete [ 6596.110494] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:34:46 (1788359686) [ 6600.242824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.451691] LustreError: 447796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.463492] LustreError: 447796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6600.494339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.902151] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6605.794239] LustreError: 447795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.890629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.127381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6610.683063] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6614.862879] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6617.974780] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6634.917267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.088864] Lustre: Found index 0 for lustre-OST0000, updating log [ 6635.219704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6638.973304] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6643.994906] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.506476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6653.093793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.105738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.111064] Lustre: Skipped 1 previous similar message [ 6658.253406] Lustre: server umount lustre-OST0000 complete [ 6672.248907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.252309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.262795] Lustre: Skipped 1 previous similar message [ 6672.271046] Lustre: Skipped 2 previous similar messages [ 6677.473968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.479891] Lustre: Skipped 2 previous similar messages [ 6678.492590] Lustre: server umount lustre-MDT0000 complete [ 6680.550935] LustreError: 449090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.570079] LustreError: 449090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6680.839168] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359770 with bad export cookie 13128585358495034906 [ 6680.841033] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.850078] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6681.059090] Lustre: server umount lustre-MDT0001 complete [ 6686.197130] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:36:16 (1788359776) [ 6694.111736] LNet: 1 local NIs in recovery (showing 1): 192.168.203.142@tcp [ 7276.393706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.595827] LustreError: 454198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.633446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.278495] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7281.633902] LustreError: 454199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.872781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.043953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7283.819054] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7286.806620] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7288.987000] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7304.997511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.126967] Lustre: Found index 0 for lustre-OST0000, updating log [ 7305.260382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.704420] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7310.680896] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7314.403533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7314.404377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.413237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7317.586133] Lustre: server umount lustre-OST0000 complete [ 7320.100816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.102226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.111346] Lustre: Skipped 1 previous similar message [ 7322.208684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7322.212057] Lustre: Skipped 4 previous similar messages [ 7326.305301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.309758] Lustre: Skipped 7 previous similar messages [ 7333.343141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.417800] Lustre: server umount lustre-MDT0000 complete [ 7333.474054] LustreError: 455438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.481823] LustreError: 455438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7334.111258] LNet: 1 local NIs in recovery (showing 1): 192.168.203.142@tcp [ 7334.760067] LustreError: 456081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360424 with bad export cookie 13128585358495035963 [ 7334.762370] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7334.766723] LustreError: 456081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7334.879684] Lustre: server umount lustre-MDT0001 complete [ 7353.188857] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7356.930937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7370.448222] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:47:40 (1788360460) [ 7377.556618] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7378.815449] Key type lgssc unregistered [ 7385.120134] LNet: 460830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7385.125815] LNetError: 460830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7435.301318] LNet: Removed LNI 192.168.203.142@tcp [ 7435.666814] Key type .llcrypt unregistered [ 7435.668685] Key type ._llcrypt unregistered [ 7443.292616] Key type ._llcrypt registered [ 7443.295588] Key type .llcrypt registered [ 7443.341791] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7443.833047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7443.841235] alg: No test for adler32 (adler32-zlib) [ 7444.709444] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 7444.830926] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7446.423246] Key type lgssc registered [ 7446.888506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7448.607650] LNet: Added LNI 192.168.203.142@tcp100 [8/256/0/180] [ 7448.942623] LNet: Added LNI 192.168.203.142@tcp101 [8/256/0/180] [ 7449.277753] LNet: Added LNI 192.168.203.142@tcp102 [8/256/0/180] [ 7449.581700] LNet: Added LNI 192.168.203.142@tcp103 [8/256/0/180] [ 7449.901359] LNet: Added LNI 192.168.203.142@tcp104 [8/256/0/180] [ 7450.225629] LNet: Added LNI 192.168.203.142@tcp105 [8/256/0/180] [ 7450.540456] LNet: Added LNI 192.168.203.142@tcp106 [8/256/0/180] [ 7450.860295] LNet: Added LNI 192.168.203.142@tcp107 [8/256/0/180] [ 7451.205404] LNet: Added LNI 192.168.203.142@tcp108 [8/256/0/180] [ 7451.537566] LNet: Added LNI 192.168.203.142@tcp109 [8/256/0/180] [ 7451.867752] LNet: Added LNI 192.168.203.142@tcp110 [8/256/0/180] [ 7452.208205] LNet: Added LNI 192.168.203.142@tcp111 [8/256/0/180] [ 7452.539710] LNet: Added LNI 192.168.203.142@tcp112 [8/256/0/180] [ 7452.872689] LNet: Added LNI 192.168.203.142@tcp113 [8/256/0/180] [ 7453.189439] LNet: Added LNI 192.168.203.142@tcp114 [8/256/0/180] [ 7453.503296] LNet: Added LNI 192.168.203.142@tcp115 [8/256/0/180] [ 7453.851691] LNet: Added LNI 192.168.203.142@tcp116 [8/256/0/180] [ 7454.181945] LNet: Added LNI 192.168.203.142@tcp117 [8/256/0/180] [ 7454.527279] LNet: Added LNI 192.168.203.142@tcp118 [8/256/0/180] [ 7454.881630] LNet: Added LNI 192.168.203.142@tcp119 [8/256/0/180] [ 7455.212632] LNet: Added LNI 192.168.203.142@tcp120 [8/256/0/180] [ 7455.537182] LNet: Added LNI 192.168.203.142@tcp121 [8/256/0/180] [ 7455.868532] LNet: Added LNI 192.168.203.142@tcp122 [8/256/0/180] [ 7456.202213] LNet: Added LNI 192.168.203.142@tcp123 [8/256/0/180] [ 7456.536637] LNet: Added LNI 192.168.203.142@tcp124 [8/256/0/180] [ 7456.861760] LNet: Added LNI 192.168.203.142@tcp125 [8/256/0/180] [ 7457.204303] LNet: Added LNI 192.168.203.142@tcp126 [8/256/0/180] [ 7457.562317] LNet: Added LNI 192.168.203.142@tcp127 [8/256/0/180] [ 7457.915845] LNet: Added LNI 192.168.203.142@tcp128 [8/256/0/180] [ 7458.281749] LNet: Added LNI 192.168.203.142@tcp129 [8/256/0/180] [ 7458.633362] LNet: Added LNI 192.168.203.142@tcp130 [8/256/0/180] [ 7458.956973] LNet: Added LNI 192.168.203.142@tcp131 [8/256/0/180] [ 7459.295860] LNet: Added LNI 192.168.203.142@tcp132 [8/256/0/180] [ 7459.615292] LNet: Added LNI 192.168.203.142@tcp133 [8/256/0/180] [ 7459.928075] LNet: Added LNI 192.168.203.142@tcp134 [8/256/0/180] [ 7460.222979] LNet: Added LNI 192.168.203.142@tcp135 [8/256/0/180] [ 7462.531115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7462.539328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.582570] Lustre: 463838:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.142@tcp: can't add NID 192.168.203.142@tcp130: rc = -75 [ 7463.645754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7463.658331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7463.694662] Lustre: lustre-MDT0000: new disk, initializing [ 7463.719813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7463.727375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7465.090071] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7469.250699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.304211] Lustre: 463856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7469.329902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7469.332587] Lustre: Skipped 1 previous similar message [ 7469.368797] Lustre: lustre-MDT0001: new disk, initializing [ 7469.381793] Lustre: 464655:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.142@tcp: can't add NID 192.168.203.142@tcp130: rc = -75 [ 7469.385585] Lustre: 464655:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7469.428760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7469.444649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7469.449499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7470.923282] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7474.960620] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7477.180224] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.239482] Lustre: 465463:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.142@tcp: can't add NID 192.168.203.142@tcp130: rc = -75 [ 7477.246908] Lustre: 465463:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7479.692098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.808130] Lustre: lustre-OST0000: new disk, initializing [ 7479.810828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7479.814165] Lustre: 465826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7479.853820] Lustre: 465831:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.142@tcp: can't add NID 192.168.203.142@tcp130: rc = -75 [ 7479.858622] Lustre: 465831:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7479.880724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.123282] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7486.304443] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7487.480647] Lustre: 466599:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.142@tcp: can't add NID 192.168.203.142@tcp130: rc = -75 [ 7487.484665] Lustre: 466599:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7487.509816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7487.514873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7487.565421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7492.579085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7492.581233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.582780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7492.592542] Lustre: Skipped 1 previous similar message [ 7492.599438] Lustre: Skipped 1 previous similar message [ 7495.265766] Lustre: server umount lustre-OST0000 complete [ 7497.697466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.702121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.705051] Lustre: Skipped 1 previous similar message [ 7497.717712] Lustre: Skipped 1 previous similar message [ 7502.818246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7502.839613] Lustre: Skipped 1 previous similar message [ 7503.401356] Lustre: server umount lustre-MDT0000 complete [ 7506.399169] LustreError: 464653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360596 with bad export cookie 839956634251122000 [ 7506.406303] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.409783] LustreError: 464653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7506.777325] Lustre: server umount lustre-MDT0001 complete [ 7517.290112] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7519.289362] Key type lgssc unregistered [ 7519.524883] LNet: 467737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7519.531767] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.543188] LNet: Removed LNI 192.168.203.142@tcp [ 7520.546748] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.552567] LNet: Removed LNI 192.168.203.142@tcp100 [ 7521.567178] LNet: Removed LNI 192.168.203.142@tcp101 [ 7521.572222] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7521.577921] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7521.582367] LNet: Removed LNI 192.168.203.142@tcp102 [ 7522.592066] LNet: Removed LNI 192.168.203.142@tcp103 [ 7523.615358] LNet: Removed LNI 192.168.203.142@tcp104 [ 7523.618975] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7523.623732] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7523.632205] LNet: Removed LNI 192.168.203.142@tcp105 [ 7524.706087] LNet: Removed LNI 192.168.203.142@tcp106 [ 7525.727398] LNet: Removed LNI 192.168.203.142@tcp107 [ 7526.751676] LNet: Removed LNI 192.168.203.142@tcp108 [ 7527.778103] LNet: Removed LNI 192.168.203.142@tcp109 [ 7527.808049] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7527.815112] LNetError: 467737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7527.822451] LNet: Removed LNI 192.168.203.142@tcp110 [ 7528.863162] LNet: Removed LNI 192.168.203.142@tcp111 [ 7529.887177] LNet: Removed LNI 192.168.203.142@tcp112 [ 7529.905171] LNet: Removed LNI 192.168.203.142@tcp113 [ 7529.918489] LNet: Removed LNI 192.168.203.142@tcp114 [ 7529.930236] LNet: Removed LNI 192.168.203.142@tcp115 [ 7529.941493] LNet: Removed LNI 192.168.203.142@tcp116 [ 7529.943648] LNet: Removed LNI 192.168.203.142@tcp117 [ 7529.954672] LNet: Removed LNI 192.168.203.142@tcp118 [ 7529.959504] LNet: Removed LNI 192.168.203.142@tcp119 [ 7529.964955] LNet: Removed LNI 192.168.203.142@tcp120 [ 7529.973179] LNet: Removed LNI 192.168.203.142@tcp121 [ 7529.975260] LNet: Removed LNI 192.168.203.142@tcp122 [ 7529.985411] LNet: Removed LNI 192.168.203.142@tcp123 [ 7529.996505] LNet: Removed LNI 192.168.203.142@tcp124 [ 7529.998626] LNet: Removed LNI 192.168.203.142@tcp125 [ 7530.012693] LNet: Removed LNI 192.168.203.142@tcp126 [ 7530.018429] LNet: Removed LNI 192.168.203.142@tcp127 [ 7530.025302] LNet: Removed LNI 192.168.203.142@tcp128 [ 7530.031288] LNet: Removed LNI 192.168.203.142@tcp129 [ 7530.044209] LNet: Removed LNI 192.168.203.142@tcp130 [ 7530.046411] LNet: Removed LNI 192.168.203.142@tcp131 [ 7530.066229] LNet: Removed LNI 192.168.203.142@tcp132 [ 7530.071245] LNet: Removed LNI 192.168.203.142@tcp133 [ 7530.083318] LNet: Removed LNI 192.168.203.142@tcp134 [ 7530.124814] LNet: Removed LNI 192.168.203.142@tcp135 [ 7531.269843] Key type .llcrypt unregistered [ 7531.271445] Key type ._llcrypt unregistered [ 7560.468963] Key type ._llcrypt registered [ 7560.471979] Key type .llcrypt registered [ 7560.739293] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7585.087642] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7586.741867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7587.171916] alg: No test for adler32 (adler32-zlib) [ 7588.289926] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 7588.691399] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7590.447170] Key type lgssc registered [ 7591.651747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7621.087870] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:51:50 (1788360710) [ 7633.885102] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7635.932265] Key type lgssc unregistered [ 7636.167695] LNet: 471831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7636.178201] LNetError: 471831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7636.191124] LNet: Removed LNI 192.168.203.142@tcp [ 7636.797878] Key type .llcrypt unregistered [ 7636.799936] Key type ._llcrypt unregistered [ 7648.776537] Key type ._llcrypt registered [ 7648.781734] Key type .llcrypt registered [ 7648.860697] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7649.505496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7649.555086] alg: No test for adler32 (adler32-zlib) [ 7650.558249] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 7650.768654] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7652.391323] Key type lgssc registered [ 7653.221332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7661.264233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7661.290624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.612169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7662.655107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7662.755811] Lustre: lustre-MDT0000: new disk, initializing [ 7662.873779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.905082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7666.828863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7675.551900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.660958] Lustre: 473023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7675.711165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7675.715343] Lustre: Skipped 1 previous similar message [ 7675.794438] Lustre: lustre-MDT0001: new disk, initializing [ 7675.884414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7675.908704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7675.923203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7679.689150] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7687.303854] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7691.702555] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7696.832676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.066494] Lustre: lustre-OST0000: new disk, initializing [ 7697.070498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7697.074508] Lustre: 474991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7697.144632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.891570] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7705.689034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7705.699163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7705.768263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7709.496867] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7712.526404] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing lnet_if_list [ 7716.866323] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7716.901175] LNet: Added LNI 192.168.203.142@tcp10 [8/256/0/180] [ 7722.126827] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7722.179260] LNet: Added LNI 192.168.203.142@tcp11 [8/256/0/180] [ 7725.743616] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7725.767498] LNet: Added LNI 192.168.203.142@tcp12 [8/256/0/180] [ 7729.117588] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7729.138720] LNet: Added LNI 192.168.203.142@tcp13 [8/256/0/180] [ 7732.788963] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7732.814751] LNet: Added LNI 192.168.203.142@tcp14 [8/256/0/180] [ 7737.288168] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7737.316265] LNet: Added LNI 192.168.203.142@tcp15 [8/256/0/180] [ 7740.964577] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7740.989810] LNet: Added LNI 192.168.203.142@tcp16 [8/256/0/180] [ 7744.975843] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7745.002051] LNet: Added LNI 192.168.203.142@tcp17 [8/256/0/180] [ 7749.456508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7749.483369] LNet: Added LNI 192.168.203.142@tcp18 [8/256/0/180] [ 7754.359536] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7754.382733] LNet: Added LNI 192.168.203.142@tcp19 [8/256/0/180] [ 7758.541764] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7758.577749] LNet: Added LNI 192.168.203.142@tcp20 [8/256/0/180] [ 7762.802413] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7762.827984] LNet: Added LNI 192.168.203.142@tcp21 [8/256/0/180] [ 7767.685344] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7767.717797] LNet: Added LNI 192.168.203.142@tcp22 [8/256/0/180] [ 7771.863525] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7771.917920] LNet: Added LNI 192.168.203.142@tcp23 [8/256/0/180] [ 7777.137703] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7777.159044] LNet: Added LNI 192.168.203.142@tcp24 [8/256/0/180] [ 7782.124530] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7782.154611] LNet: Added LNI 192.168.203.142@tcp25 [8/256/0/180] [ 7787.419899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7787.453806] LNet: Added LNI 192.168.203.142@tcp26 [8/256/0/180] [ 7792.890316] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7792.932647] LNet: Added LNI 192.168.203.142@tcp27 [8/256/0/180] [ 7798.029736] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7798.053842] LNet: Added LNI 192.168.203.142@tcp28 [8/256/0/180] [ 7802.950775] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7802.970805] LNet: Added LNI 192.168.203.142@tcp29 [8/256/0/180] [ 7807.499276] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7807.525739] LNet: Added LNI 192.168.203.142@tcp30 [8/256/0/180] [ 7812.530658] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7812.570677] LNet: Added LNI 192.168.203.142@tcp31 [8/256/0/180] [ 7817.656752] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7817.679564] LNet: Added LNI 192.168.203.142@tcp32 [8/256/0/180] [ 7823.240523] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7823.273904] LNet: Added LNI 192.168.203.142@tcp33 [8/256/0/180] [ 7828.448688] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7828.475262] LNet: Added LNI 192.168.203.142@tcp34 [8/256/0/180] [ 7833.705442] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7833.730133] LNet: Added LNI 192.168.203.142@tcp35 [8/256/0/180] [ 7839.831602] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7839.871051] LNet: Added LNI 192.168.203.142@tcp36 [8/256/0/180] [ 7845.870460] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7845.898700] LNet: Added LNI 192.168.203.142@tcp37 [8/256/0/180] [ 7852.507931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7852.545333] LNet: Added LNI 192.168.203.142@tcp38 [8/256/0/180] [ 7858.749458] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7858.774410] LNet: Added LNI 192.168.203.142@tcp39 [8/256/0/180] [ 7864.849286] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7864.888243] LNet: Added LNI 192.168.203.142@tcp40 [8/256/0/180] [ 7871.018517] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7871.055105] LNet: Added LNI 192.168.203.142@tcp41 [8/256/0/180] [ 7878.146115] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7878.179353] LNet: Added LNI 192.168.203.142@tcp42 [8/256/0/180] [ 7885.260216] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7885.294070] LNet: Added LNI 192.168.203.142@tcp43 [8/256/0/180] [ 7892.713373] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7892.763662] LNet: Added LNI 192.168.203.142@tcp44 [8/256/0/180] [ 7899.699777] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7899.738987] LNet: Added LNI 192.168.203.142@tcp45 [8/256/0/180] [ 7906.372647] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7906.394487] LNet: Added LNI 192.168.203.142@tcp46 [8/256/0/180] [ 7912.840659] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7912.862562] LNet: Added LNI 192.168.203.142@tcp47 [8/256/0/180] [ 7919.911855] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7919.943506] LNet: Added LNI 192.168.203.142@tcp48 [8/256/0/180] [ 7927.318241] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7927.341712] LNet: Added LNI 192.168.203.142@tcp49 [8/256/0/180] [ 7934.200955] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7934.233213] LNet: Added LNI 192.168.203.142@tcp50 [8/256/0/180] [ 7951.843055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.852405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7951.858487] Lustre: Skipped 2 previous similar messages [ 7951.872541] Lustre: Skipped 2 previous similar messages [ 7956.967351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7956.981062] Lustre: Skipped 2 previous similar messages [ 7962.084315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7962.095026] Lustre: Skipped 2 previous similar messages [ 7963.103198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7963.244753] Lustre: server umount lustre-MDT0000 complete [ 7966.174644] LustreError: 474990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361056 with bad export cookie 7832873151046048265 [ 7966.176089] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7966.196486] LustreError: 474990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7966.471244] Lustre: server umount lustre-MDT0001 complete [ 7980.580138] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7983.583197] Lustre: 472294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361057/real 1788361057] req@0000000074c0d79c x1875232151455872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788361073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7983.609925] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.202446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.743154] Lustre: 472294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361061/real 1788361061] req@00000000e4da9b24 x1875232151456000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788361077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7991.839610] LustreError: 472293:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000872b08be x1875232151456768/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7992.091608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.119546] Lustre: 472296:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361066/real 1788361066] req@000000009db73d1f x1875232151456384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788361082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7993.125950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7993.157930] Lustre: 472296:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7995.523291] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7996.895240] Lustre: 472294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361071/real 1788361071] req@00000000a8ccad27 x1875232151456512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788361087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7998.437711] LustreError: 499204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.459143] LustreError: 499204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8002.204891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.533794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.855656] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8007.660526] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8010.914417] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8015.125566] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8033.252437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.253227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.261748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8033.267866] Lustre: Skipped 1 previous similar message [ 8034.693271] Lustre: server umount lustre-OST0000 complete [ 8038.374316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.374347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8038.377589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8038.377596] Lustre: Skipped 1 previous similar message [ 8038.385768] Lustre: Skipped 1 previous similar message [ 8043.642691] Lustre: server umount lustre-MDT0000 complete [ 8046.362080] LustreError: 473017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361136 with bad export cookie 7832873151046050792 [ 8046.363757] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8046.370885] LustreError: 473017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8046.559156] Lustre: server umount lustre-MDT0001 complete [ 8054.407181] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8056.678543] Key type lgssc unregistered [ 8056.983269] LNet: 501981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8056.990565] LNetError: 501981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.995702] LNet: Removed LNI 192.168.203.142@tcp [ 8056.999095] LNet: Removed LNI 192.168.203.142@tcp10 [ 8057.002492] LNet: Removed LNI 192.168.203.142@tcp11 [ 8057.006737] LNet: Removed LNI 192.168.203.142@tcp12 [ 8057.010827] LNet: Removed LNI 192.168.203.142@tcp13 [ 8057.014795] LNet: Removed LNI 192.168.203.142@tcp14 [ 8057.016928] LNet: Removed LNI 192.168.203.142@tcp15 [ 8057.023856] LNet: Removed LNI 192.168.203.142@tcp16 [ 8057.030270] LNet: Removed LNI 192.168.203.142@tcp17 [ 8057.033614] LNet: Removed LNI 192.168.203.142@tcp18 [ 8057.038379] LNet: Removed LNI 192.168.203.142@tcp19 [ 8057.041344] LNet: Removed LNI 192.168.203.142@tcp20 [ 8057.043263] LNet: Removed LNI 192.168.203.142@tcp21 [ 8057.047351] LNet: Removed LNI 192.168.203.142@tcp22 [ 8057.055770] LNet: Removed LNI 192.168.203.142@tcp23 [ 8057.061102] LNet: Removed LNI 192.168.203.142@tcp24 [ 8057.064547] LNet: Removed LNI 192.168.203.142@tcp25 [ 8057.066989] LNet: Removed LNI 192.168.203.142@tcp26 [ 8057.069864] LNet: Removed LNI 192.168.203.142@tcp27 [ 8057.073575] LNet: Removed LNI 192.168.203.142@tcp28 [ 8057.079285] LNet: Removed LNI 192.168.203.142@tcp29 [ 8057.085066] LNet: Removed LNI 192.168.203.142@tcp30 [ 8057.087796] LNet: Removed LNI 192.168.203.142@tcp31 [ 8057.092091] LNet: Removed LNI 192.168.203.142@tcp32 [ 8057.094801] LNet: Removed LNI 192.168.203.142@tcp33 [ 8057.106623] LNet: Removed LNI 192.168.203.142@tcp34 [ 8057.109976] LNet: Removed LNI 192.168.203.142@tcp35 [ 8057.112772] LNet: Removed LNI 192.168.203.142@tcp36 [ 8057.116048] LNet: Removed LNI 192.168.203.142@tcp37 [ 8057.119259] LNet: Removed LNI 192.168.203.142@tcp38 [ 8057.122156] LNet: Removed LNI 192.168.203.142@tcp39 [ 8057.124749] LNet: Removed LNI 192.168.203.142@tcp40 [ 8057.130937] LNet: Removed LNI 192.168.203.142@tcp41 [ 8057.132779] LNet: Removed LNI 192.168.203.142@tcp42 [ 8057.141588] LNet: Removed LNI 192.168.203.142@tcp43 [ 8057.144479] LNet: Removed LNI 192.168.203.142@tcp44 [ 8057.149065] LNet: Removed LNI 192.168.203.142@tcp45 [ 8057.151244] LNet: Removed LNI 192.168.203.142@tcp46 [ 8057.153499] LNet: Removed LNI 192.168.203.142@tcp47 [ 8057.155662] LNet: Removed LNI 192.168.203.142@tcp48 [ 8057.157955] LNet: Removed LNI 192.168.203.142@tcp49 [ 8057.167935] LNet: Removed LNI 192.168.203.142@tcp50 [ 8057.677868] Key type .llcrypt unregistered [ 8057.681865] Key type ._llcrypt unregistered [ 8074.871419] Key type ._llcrypt registered [ 8074.875535] Key type .llcrypt registered [ 8075.036432] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8087.214746] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8088.210508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8088.225263] alg: No test for adler32 (adler32-zlib) [ 8089.231585] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 8089.508264] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 8091.159253] Key type lgssc registered [ 8092.094660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.359155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.388801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.603841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8121.635283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8121.682286] Lustre: lustre-MDT0000: new disk, initializing [ 8121.752556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.767562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8125.032903] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8133.158513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.250477] Lustre: 505166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8133.315193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8133.319528] Lustre: Skipped 1 previous similar message [ 8133.398466] Lustre: lustre-MDT0001: new disk, initializing [ 8133.487280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8133.533837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8133.544429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8136.748317] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8144.324873] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8148.435759] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8153.998297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.145616] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8154.197804] Lustre: lustre-OST0000: new disk, initializing [ 8154.201803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8154.208085] Lustre: 507179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8154.280939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.425581] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8162.851921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8162.866536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8162.930692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8164.394309] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8168.978289] Lustre: server umount lustre-OST0000 complete [ 8173.536750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.538417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.557471] Lustre: Skipped 1 previous similar message [ 8173.565715] Lustre: Skipped 1 previous similar message [ 8178.661319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8178.666681] Lustre: Skipped 1 previous similar message [ 8183.776361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.784747] Lustre: Skipped 1 previous similar message [ 8186.335585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8186.468116] Lustre: server umount lustre-MDT0000 complete [ 8188.821374] LustreError: 505157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361278 with bad export cookie 8170202482084766203 [ 8188.823776] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.828948] LustreError: 505157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8198.226405] Lustre: server umount lustre-MDT0001 complete [ 8205.136252] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8206.941544] Key type lgssc unregistered [ 8207.114775] LNet: 509096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8207.120159] LNetError: 509096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8207.129827] LNet: Removed LNI 192.168.203.142@tcp [ 8207.680384] Key type .llcrypt unregistered [ 8207.682156] Key type ._llcrypt unregistered [ 8220.888811] Key type ._llcrypt registered [ 8220.891671] Key type .llcrypt registered [ 8220.966669] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8221.658577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8221.676763] alg: No test for adler32 (adler32-zlib) [ 8222.670775] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 8222.818958] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 8224.455265] Key type lgssc registered [ 8225.372731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8230.465467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8230.485465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.749762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.545481] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8240.099996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.337557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8243.038573] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8247.933294] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8251.755280] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.199796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.443343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.491128] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8265.718265] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8272.707438] loop0: detected capacity change from 0 to 204800000 [ 8272.824301] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8272.836486] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8272.848947] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8276.825385] loop0: detected capacity change from 0 to 204800000 [ 8276.925267] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8276.935895] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8276.948375] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8280.346419] loop0: detected capacity change from 0 to 204800000 [ 8283.639171] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8283.748651] Lustre: fs15246-OST0000: new disk, initializing [ 8283.754472] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8283.763659] Lustre: 513933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8283.800474] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8287.652838] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8293.746571] loop1: detected capacity change from 0 to 204800000 [ 8297.108976] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.195536] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8297.245851] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8297.299669] Lustre: fs15246-MDT0000: new disk, initializing [ 8297.401291] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.423402] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8297.501848] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8298.528123] Lustre: 515054:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b2a7b9fd x1875232751131008/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788361404 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8300.026554] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8307.619340] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8307.633634] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8307.648255] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8314.853248] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.866374] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8319.485990] Lustre: server umount fs15246-MDT0000 complete [ 8319.981109] LustreError: 510340:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8320.003662] LustreError: 510340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8324.956752] Lustre: server umount fs15246-OST0000 complete [ 8332.257813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.267133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.277549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8334.305383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.312301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.320120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.639659] Lustre: server umount lustre-OST0000 complete [ 8340.452588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.457218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.460077] Lustre: Skipped 1 previous similar message [ 8340.471700] Lustre: Skipped 3 previous similar messages [ 8345.573215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.579827] Lustre: Skipped 1 previous similar message [ 8354.271994] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8354.358633] Lustre: server umount lustre-MDT0000 complete [ 8355.810137] LustreError: 510344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.817732] LustreError: 510344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8356.507526] LustreError: 510324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361446 with bad export cookie 6043944316877588030 [ 8356.509861] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8356.513958] LustreError: 510324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8356.690472] Lustre: server umount lustre-MDT0001 complete [ 8362.897171] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8364.618797] Key type lgssc unregistered [ 8364.819951] LNet: 518151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8364.825711] LNetError: 518151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.834792] LNet: Removed LNI 192.168.203.142@tcp [ 8365.357393] Key type .llcrypt unregistered [ 8365.360874] Key type ._llcrypt unregistered [ 8376.238602] Key type ._llcrypt registered [ 8376.240816] Key type .llcrypt registered [ 8376.323697] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8376.725929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8376.734084] alg: No test for adler32 (adler32-zlib) [ 8377.675770] Lustre: Lustre: Build Version: 2.17.55_23_g2958cf4 [ 8377.809931] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 8379.415755] Key type lgssc registered [ 8380.109826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.378977] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8409.258945] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8430.676796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8430.686955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.839830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8431.863727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8431.914814] Lustre: lustre-MDT0000: new disk, initializing [ 8431.976416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.985697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8434.479443] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8441.236692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.330474] Lustre: 522589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8441.381134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8441.384285] Lustre: Skipped 1 previous similar message [ 8441.453734] Lustre: lustre-MDT0001: new disk, initializing [ 8441.520624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8441.544728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8441.549981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8444.117592] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8449.577366] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8452.856513] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8456.815524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8457.013092] Lustre: lustre-OST0000: new disk, initializing [ 8457.017220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8457.022878] Lustre: 524557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8457.074761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.674696] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8465.957655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.964184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8466.078043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8467.018794] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.635696] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8469.786937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8472.603881] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8472.771234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8476.129380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.137950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.154295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.645446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.646757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.660884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8480.149260] Lustre: server umount lustre-OST0000 complete [ 8482.785215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8482.790653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.802579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8486.880948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8486.881353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.569819] Lustre: server umount lustre-MDT0000 complete [ 8490.762115] LustreError: 522582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361580 with bad export cookie 8090319240471426 [ 8490.765167] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8490.770920] LustreError: 522582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8490.954977] Lustre: server umount lustre-MDT0001 complete [ 8504.753771] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8510.269488] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8527.252637] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8401 sec ========== 11:06:57 (1788361617) [ 8528.080120] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:06:58 (1788361618) === [ 8529.746651] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:06:59 (1788361619) === [ 8549.483184] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8550.958987] Key type lgssc unregistered [ 8551.120739] LNet: 531192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8551.126158] LNetError: 531192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8551.135106] LNet: Removed LNI 192.168.203.142@tcp [ 8551.571254] Key type .llcrypt unregistered [ 8551.573171] Key type ._llcrypt unregistered