[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475459672 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003252] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008611] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011098] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013052] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.023366] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028189] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031109] *** VALIDATE tmpfs *** [ 0.032450] *** VALIDATE proc *** [ 0.033263] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041534] debug: unmapping init [mem 0xffffffff9ae59000-0xffffffff9ae60fff] [ 0.044194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051025] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.054195] rcu: Hierarchical SRCU implementation. [ 0.056343] smp: Bringing up secondary CPUs ... [ 0.057558] x86: Booting SMP configuration: [ 0.058035] .... node #0, CPUs: #1 #2 #3 [ 0.061737] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230679] node 0 deferred pages initialised in 164ms [ 0.233335] devtmpfs: initialized [ 0.234313] x86/mm: Memory block size: 128MB [ 0.238009] gcov: version magic: 0x41383552 [ 0.241329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247349] pinctrl core: initialized pinctrl subsystem [ 0.249174] [ 0.249729] ************************************************************* [ 0.252016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253011] ** ** [ 0.255014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258015] ** ** [ 0.260014] ** This means that this kernel is built to expose internal ** [ 0.262012] ** IOMMU data structures, which may compromise security on ** [ 0.264016] ** your system. ** [ 0.266011] ** ** [ 0.269018] ** If you see this message and you are not debugging the ** [ 0.271014] ** kernel, report this immediately to your vendor! ** [ 0.273015] ** ** [ 0.275016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277015] ************************************************************* [ 0.279634] NET: Registered protocol family 16 [ 0.280412] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288173] cpuidle: using governor menu [ 0.290000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.292563] PCI: Using configuration type 1 for base access [ 0.294151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.304192] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307082] cryptd: max_cpu_qlen set to 1000 [ 0.309216] ACPI: Added _OSI(Module Device) [ 0.311022] ACPI: Added _OSI(Processor Device) [ 0.313042] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.319754] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.327319] ACPI: Interpreter enabled [ 0.329090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.330039] ACPI: Using IOAPIC for interrupt routing [ 0.332167] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336536] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347627] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361549] acpiphp: Slot [2] registered [ 0.364183] acpiphp: Slot [5] registered [ 0.366178] acpiphp: Slot [6] registered [ 0.367162] acpiphp: Slot [7] registered [ 0.369171] acpiphp: Slot [8] registered [ 0.371180] acpiphp: Slot [9] registered [ 0.372154] acpiphp: Slot [10] registered [ 0.374154] acpiphp: Slot [3] registered [ 0.376134] acpiphp: Slot [4] registered [ 0.378166] acpiphp: Slot [11] registered [ 0.379137] acpiphp: Slot [12] registered [ 0.381171] acpiphp: Slot [13] registered [ 0.383141] acpiphp: Slot [14] registered [ 0.384089] acpiphp: Slot [15] registered [ 0.385110] acpiphp: Slot [16] registered [ 0.386107] acpiphp: Slot [17] registered [ 0.388155] acpiphp: Slot [18] registered [ 0.390126] acpiphp: Slot [19] registered [ 0.391171] acpiphp: Slot [20] registered [ 0.393177] acpiphp: Slot [21] registered [ 0.395167] acpiphp: Slot [22] registered [ 0.396146] acpiphp: Slot [23] registered [ 0.398150] acpiphp: Slot [24] registered [ 0.400145] acpiphp: Slot [25] registered [ 0.401135] acpiphp: Slot [26] registered [ 0.402088] acpiphp: Slot [27] registered [ 0.404146] acpiphp: Slot [28] registered [ 0.406149] acpiphp: Slot [29] registered [ 0.407151] acpiphp: Slot [30] registered [ 0.409145] acpiphp: Slot [31] registered [ 0.411082] PCI host bridge to bus 0000:00 [ 0.412026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.417033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.425030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.428214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443743] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.449462] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.452030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459523] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.462755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.469854] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.474018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.487023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.493034] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.499129] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.507021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.517021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.546337] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.560176] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.594615] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.606020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.634016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.647943] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.662019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.676019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.685583] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.692016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.698016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.724123] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.729015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.757056] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.760388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.762405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.765387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.767187] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.772242] iommu: Default domain type: Passthrough [ 0.774343] SCSI subsystem initialized [ 0.775138] ACPI: bus type USB registered [ 0.777125] usbcore: registered new interface driver usbfs [ 0.779099] usbcore: registered new interface driver hub [ 0.781100] usbcore: registered new device driver usb [ 0.783207] pps_core: LinuxPPS API ver. 1 registered [ 0.784010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787058] PTP clock support registered [ 0.789117] EDAC MC: Ver: 3.0.0 [ 0.790569] PCI: Using ACPI for IRQ routing [ 0.793310] NetLabel: Initializing [ 0.794012] NetLabel: domain hash size = 128 [ 0.796013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.797086] NetLabel: unlabeled traffic allowed by default [ 0.799151] vgaarb: loaded [ 0.801334] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.804023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811000] clocksource: Switched to clocksource kvm-clock [ 0.922974] VFS: Disk quotas dquot_6.6.0 [ 0.924718] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.927747] *** VALIDATE ramfs *** [ 0.929308] *** VALIDATE hugetlbfs *** [ 0.931186] pnp: PnP ACPI init [ 0.933909] pnp: PnP ACPI: found 6 devices [ 0.953844] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.957757] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.960469] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.963039] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966048] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968576] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.971882] NET: Registered protocol family 2 [ 0.974698] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.980150] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.984072] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.990399] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.994856] TCP: Hash tables configured (established 65536 bind 65536) [ 0.998301] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.002063] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003951] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006827] NET: Registered protocol family 1 [ 1.009127] RPC: Registered named UNIX socket transport module. [ 1.012383] RPC: Registered udp transport module. [ 1.014142] RPC: Registered tcp transport module. [ 1.015831] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.019687] NET: Registered protocol family 44 [ 1.021538] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.023595] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.025959] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.029768] PCI: CLS 0 bytes, default 64 [ 1.031653] Unpacking initramfs... [ 2.446163] debug: unmapping init [mem 0xffff942ffcc54000-0xffff942ffffbffff] [ 2.450483] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.453138] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.456436] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.961882] Initialise system trusted keyrings [ 2.963678] Key type blacklist registered [ 2.967312] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.978937] zbud: loaded [ 2.982499] *** VALIDATE nfs *** [ 2.983952] *** VALIDATE nfs4 *** [ 2.986051] pstore: using deflate compression [ 2.990282] Platform Keyring initialized [ 3.112671] NET: Registered protocol family 38 [ 3.114528] Key type asymmetric registered [ 3.116232] Asymmetric key parser 'x509' registered [ 3.118238] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.122116] io scheduler mq-deadline registered [ 3.124163] io scheduler kyber registered [ 3.126056] io scheduler bfq registered [ 3.128400] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.132431] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.135232] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.137713] ACPI: Power Button [PWRF] [ 3.143233] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.149535] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.163278] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.171341] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.185580] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.214358] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.243019] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.250192] Non-volatile memory driver v1.3 [ 3.252235] Linux agpgart interface v0.103 [ 3.284237] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.288489] vda: detected capacity change from 0 to 76521472 [ 3.304414] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.307642] vdb: detected capacity change from 0 to 1073741824 [ 3.322397] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325930] vdc: detected capacity change from 0 to 2621440000 [ 3.341657] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.344588] vdd: detected capacity change from 0 to 2621440000 [ 3.359674] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362763] vde: detected capacity change from 0 to 4294967296 [ 3.378369] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.382190] vdf: detected capacity change from 0 to 4294967296 [ 3.388946] libphy: Fixed MDIO Bus: probed [ 3.395050] usbcore: registered new interface driver usbserial_generic [ 3.397900] usbserial: USB Serial support registered for generic [ 3.400787] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.406541] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.408507] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.411488] mousedev: PS/2 mouse device common for all mice [ 3.414562] rtc_cmos 00:05: RTC can wake from S4 [ 3.416885] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.417763] rtc_cmos 00:05: registered as rtc0 [ 3.422498] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.424781] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.426618] intel_pstate: CPU model not supported [ 3.432831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.437373] hid: raw HID events driver (C) Jiri Kosina [ 3.438807] usbcore: registered new interface driver usbhid [ 3.440210] usbhid: USB HID core driver [ 3.441328] drop_monitor: Initializing network drop monitor service [ 3.443368] Initializing XFRM netlink socket [ 3.444755] NET: Registered protocol family 10 [ 3.446840] Segment Routing with IPv6 [ 3.447895] NET: Registered protocol family 17 [ 3.449249] mpls_gso: MPLS GSO support [ 3.453658] RAS: Correctable Errors collector initialized. [ 3.455978] AVX version of gcm_enc/dec engaged. [ 3.457736] AES CTR mode by8 optimization enabled [ 3.541495] sched_clock: Marking stable (3541468623, 0)->(4477628761, -936160138) [ 3.545285] registered taskstats version 1 [ 3.547324] Loading compiled-in X.509 certificates [ 3.549409] zswap: loaded using pool lzo/zbud [ 3.573120] Key type big_key registered [ 3.585057] Key type encrypted registered [ 3.586594] ima: No TPM chip found, activating TPM-bypass! [ 3.589245] ima: Allocated hash algorithm: sha1 [ 3.590924] ima: No architecture policies found [ 3.592668] evm: Initialising EVM extended attributes: [ 3.594664] evm: security.selinux [ 3.596264] evm: security.ima [ 3.597525] evm: security.capability [ 3.598927] evm: HMAC attrs: 0x1 [ 3.601513] rtc_cmos 00:05: setting system clock to 2026-08-22 07:22:16 UTC (1787383336) [ 3.608070] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.611219] debug: unmapping init [mem 0xffffffff9ab82000-0xffffffff9ae58fff] [ 3.620123] Write protecting the kernel read-only data: 28672k [ 3.624129] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.627386] debug: unmapping init [mem 0xffffffff99b14000-0xffffffff99bfffff] [ 3.669817] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679679] systemd[1]: Detected virtualization kvm. [ 3.682071] systemd[1]: Detected architecture x86-64. [ 3.684381] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.712375] systemd[1]: No hostname configured. [ 3.714369] systemd[1]: Set hostname to . [ 3.716746] random: systemd: uninitialized urandom read (16 bytes read) [ 3.719417] systemd[1]: Initializing machine ID from random generator. [ 3.759574] random: ln: uninitialized urandom read (6 bytes read) [ 3.845153] random: systemd: uninitialized urandom read (16 bytes read) [ 3.848687] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.854793] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.860651] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.448759] device-mapper: uevent: version 1.0.3 [ 4.451482] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.204312] virtio_net virtio0 ens2: renamed from eth0 [ 5.229443] random: fast init done [ 5.301972] scsi host0: ata_piix [ 5.351820] scsi host1: ata_piix [ 5.353619] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.355721] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.120573] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.104544] random: crng init done [ 10.105710] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.573580] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.730452] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.008242] SELinux: Disabled at runtime. [ 12.073079] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.083462] systemd[1]: Detected virtualization kvm. [ 12.086317] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.628533] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.633291] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.640097] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.645432] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.649056] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.657236] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.663627] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ 12.737917] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.206607] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.533393] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.608557] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.770959] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.784770] EDAC sbridge: Ver: 1.1.2 [ 15.627035] Key type dns_resolver registered [ 15.936444] NFS: Registering the id_resolver key type [ 15.938370] Key type id_resolver registered [ 15.939755] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-server login: [ 84.474479] libcfs: loading out-of-tree module taints kernel. [ 84.513500] Key type ._llcrypt registered [ 84.515845] Key type .llcrypt registered [ 84.620538] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 101.956637] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 103.462633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 103.476158] alg: No test for adler32 (adler32-zlib) [ 104.835986] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 105.648245] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 107.383502] Key type lgssc registered [ 108.980451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.587940] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.166314] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 175.365391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.413567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.835459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.884549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.987298] Lustre: lustre-MDT0000: new disk, initializing [ 177.154147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.187122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.461767] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.441168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.585898] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.639315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.642786] Lustre: Skipped 1 previous similar message [ 196.762276] Lustre: lustre-MDT0001: new disk, initializing [ 196.880521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.919988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.930131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.616498] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.896049] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.555800] hrtimer: interrupt took 8462096 ns [ 219.088306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.382326] Lustre: lustre-OST0000: new disk, initializing [ 219.385783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.391478] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.511177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.623258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.639972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.763929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.542693] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.861742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.018132] Lustre: lustre-OST0001: new disk, initializing [ 242.023756] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.037857] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.170477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.502075] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.412985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.425056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.480499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.392058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.988810] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 275.306446] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.733521] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing yml_node [ 285.249361] Lustre: DEBUG MARKER: Client: 2.17.54.114 [ 287.854342] Lustre: DEBUG MARKER: MDS: 2.17.54.114 [ 290.151278] Lustre: DEBUG MARKER: OSS: 2.17.54.114 [ 291.672187] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:27:02 EDT 2026 [ 308.135430] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 309.947845] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 327.139691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.141135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.155286] Lustre: Skipped 1 previous similar message [ 331.798057] Lustre: server umount lustre-MDT0000 complete [ 336.064860] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383668 with bad export cookie 1889975961134782607 [ 336.069229] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.080569] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 337.379908] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.380162] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.383556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.383569] Lustre: Skipped 3 previous similar messages [ 337.405336] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 337.410920] Lustre: Skipped 3 previous similar messages [ 341.408154] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.408953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.450346] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 341.458368] Lustre: Skipped 2 previous similar messages [ 342.642217] Lustre: server umount lustre-MDT0001 complete [ 348.540576] Lustre: server umount lustre-OST0000 complete [ 352.989714] Lustre: server umount lustre-OST0001 complete [ 359.852286] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 369.559779] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 401.734628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.962706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.000050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.096939] Lustre: lustre-MDT0000: new disk, initializing [ 402.198654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.226700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.837225] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 415.182103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.296645] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.358379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.363420] Lustre: Skipped 1 previous similar message [ 415.467503] Lustre: lustre-MDT0001: new disk, initializing [ 415.577833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.628511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.637953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.687375] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 428.971346] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.718440] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 441.641149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.907304] Lustre: lustre-OST0000: new disk, initializing [ 441.910796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.918892] Lustre: 17047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.015223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.975562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.989664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.120409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.240715] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 457.452037] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 462.050108] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.232416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.334055] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.505515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.993604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.994181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.002374] LustreError: Skipped 1 previous similar message [ 470.003855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.037496] Lustre: Skipped 1 previous similar message [ 474.334787] Lustre: server umount lustre-OST0000 complete [ 480.226416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.229524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.244971] Lustre: Skipped 1 previous similar message [ 480.254406] Lustre: Skipped 2 previous similar messages [ 484.981114] Lustre: server umount lustre-MDT0000 complete [ 485.346891] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.368486] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 488.906578] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383821 with bad export cookie 1889975961134784133 [ 488.908252] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.918982] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 489.256188] Lustre: server umount lustre-MDT0001 complete [ 493.793674] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:30:24 (1787383824) [ 496.949202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 499.031799] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:30:29 (1787383829) [ 506.622859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.221196] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.336906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.566923] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 512.486121] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.223058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.438299] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 533.197693] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 539.386759] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 547.573742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.856717] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 563.120502] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 575.970900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 575.971420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.979276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 575.985379] Lustre: Skipped 1 previous similar message [ 575.995362] Lustre: Skipped 1 previous similar message [ 580.270132] Lustre: server umount lustre-OST0000 complete [ 586.210058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.230071] Lustre: Skipped 1 previous similar message [ 589.730269] Lustre: server umount lustre-MDT0000 complete [ 591.328057] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.342537] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 593.503629] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383926 with bad export cookie 1889975961134785022 [ 593.506049] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.514524] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 593.928615] Lustre: server umount lustre-MDT0001 complete [ 604.713211] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 607.646504] Key type lgssc unregistered [ 607.889244] LNet: 22990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.899075] LNetError: 22990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.923065] LNet: Removed LNI 192.168.203.159@tcp [ 608.760502] Key type .llcrypt unregistered [ 608.764757] Key type ._llcrypt unregistered [ 630.145280] Key type ._llcrypt registered [ 630.147146] Key type .llcrypt registered [ 630.232260] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 631.393642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.423205] alg: No test for adler32 (adler32-zlib) [ 632.570845] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 632.979645] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 634.696511] Key type lgssc registered [ 636.266390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 647.282811] loop: module loaded [ 647.403586] loop0: detected capacity change from 0 to 204800000 [ 647.683176] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.714714] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.181284] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.038551] loop0: detected capacity change from 0 to 204800000 [ 655.199347] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.217420] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.238906] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.548831] loop0: detected capacity change from 0 to 204800000 [ 666.362222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 666.406768] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.736756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 667.827405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 667.954763] Lustre: lustre-MDT0000: new disk, initializing [ 668.044373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.074495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 673.175912] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 682.097079] loop1: detected capacity change from 0 to 204800000 [ 688.112119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.486345] Lustre: lustre-OST07c6: new disk, initializing [ 688.497941] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 688.502838] Lustre: Skipped 1 previous similar message [ 688.507306] Lustre: 26177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 688.607754] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 695.398920] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 696.923362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 696.937737] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 696.983283] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 701.491857] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 701.510866] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 701.516495] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 701.529607] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 701.537029] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 701.547442] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.331459] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 714.347554] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.351890] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.358413] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 714.367759] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 714.385751] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.432880] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 719.447565] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.464749] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.472289] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.478036] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.484252] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.493955] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 719.504756] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.516268] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 719.523164] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.531776] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.542621] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.776077] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 722.786633] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.796630] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.812315] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.820412] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.834343] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.843581] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 722.854879] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.868456] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.877534] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.886320] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.896324] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.903729] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 727.912567] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.918124] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.924415] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.929844] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.939641] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.946421] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 727.952036] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.955768] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.964398] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.970583] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.975686] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.121077] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 736.133127] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 736.146403] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 736.154737] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 736.166668] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.177905] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 736.190383] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 736.197687] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 736.203314] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 736.207863] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 736.218558] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 736.226732] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 788.983911] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.994359] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.022942] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.083765] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.815637] Lustre: server umount lustre-OST07c6 complete [ 805.181248] Lustre: server umount lustre-MDT0000 complete [ 825.920773] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 829.495723] Key type lgssc unregistered [ 829.764662] LNet: 28740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.786461] LNetError: 28740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.814365] LNet: Removed LNI 192.168.203.159@tcp [ 830.614828] Key type .llcrypt unregistered [ 830.616800] Key type ._llcrypt unregistered [ 850.727491] Key type ._llcrypt registered [ 850.730474] Key type .llcrypt registered [ 850.819176] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 851.784301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.795687] alg: No test for adler32 (adler32-zlib) [ 853.279483] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 853.651840] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 855.351728] Key type lgssc registered [ 857.364524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.364792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.410754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.879298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.420511] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 879.079885] LustreError: 30011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.095535] LustreError: 30011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 882.563422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.070802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 888.068061] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 895.772844] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.326436] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.073980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.431815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.542245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 914.349645] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 922.108711] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 942.700544] Lustre: server umount lustre-OST0000 complete [ 948.708643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.709849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.714455] Lustre: Skipped 1 previous similar message [ 953.826095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.833694] Lustre: Skipped 2 previous similar messages [ 957.088668] Lustre: server umount lustre-MDT0000 complete [ 958.950227] LustreError: 30012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.976957] LustreError: 30012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 959.968092] Lustre: 29320:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787384276/real 1787384276] req@0000000093a2cfb8 x1874208160250112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787384292 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 959.995200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.010295] LustreError: 31252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.019109] LustreError: 31252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 961.013922] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384293 with bad export cookie 12819781906128188492 [ 961.022533] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.030950] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 961.322521] Lustre: server umount lustre-MDT0001 complete [ 971.639990] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 974.788243] Key type lgssc unregistered [ 975.122301] LNet: 33749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.133238] LNetError: 33749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.162155] LNet: Removed LNI 192.168.203.159@tcp [ 976.197096] Key type .llcrypt unregistered [ 976.201293] Key type ._llcrypt unregistered [ 996.765684] Key type ._llcrypt registered [ 996.769481] Key type .llcrypt registered [ 996.859178] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 997.870558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.948127] alg: No test for adler32 (adler32-zlib) [ 999.233174] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 999.607067] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1001.311450] Key type lgssc registered [ 1002.618725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.425206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.445322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.037958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.226774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1021.412208] LustreError: 35021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.432885] LustreError: 35021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1024.926418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.381623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.125420] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1039.431555] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1045.645353] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1053.083787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1053.543456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.623767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1060.381807] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1068.608570] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1086.433285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.435416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.462356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.991338] Lustre: server umount lustre-OST0000 complete [ 1096.675250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.692329] Lustre: Skipped 1 previous similar message [ 1096.698024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.702682] Lustre: Skipped 1 previous similar message [ 1099.593233] Lustre: server umount lustre-MDT0000 complete [ 1101.813879] LustreError: 35750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.830622] LustreError: 35750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1103.751498] LustreError: 35002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384436 with bad export cookie 8434316852506586254 [ 1103.752608] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1103.761979] LustreError: 35002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1104.092895] Lustre: server umount lustre-MDT0001 complete [ 1115.406274] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1118.348711] Key type lgssc unregistered [ 1118.794674] LNet: 38577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.802121] LNetError: 38577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.816774] LNet: Removed LNI 192.168.203.159@tcp [ 1119.856965] Key type .llcrypt unregistered [ 1119.872675] Key type ._llcrypt unregistered [ 1146.362801] Key type ._llcrypt registered [ 1146.364344] Key type .llcrypt registered [ 1146.505943] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1148.256372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1148.306431] alg: No test for adler32 (adler32-zlib) [ 1149.423947] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1149.748489] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1151.511186] Key type lgssc registered [ 1152.967935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1161.239691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1161.257296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.869207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.547273] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1172.466361] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.489662] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1177.589404] LustreError: 39996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.465595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.985235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1184.447313] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1194.097598] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1200.736945] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1208.725766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.282362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.531697] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1225.113704] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1232.830057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1241.486690] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1243.116875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.131697] Lustre: Skipped 2 previous similar messages [ 1243.136537] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.160390] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1243.268246] Lustre: server umount lustre-MDT0000 complete [ 1246.561976] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.842621] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384579 with bad export cookie 638510053902429812 [ 1246.855547] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.856867] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1247.228024] Lustre: server umount lustre-MDT0001 complete [ 1259.511385] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.59@tcp arrived at 1787384592 with bad export cookie 638510053902430477 [ 1259.527954] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.987459] Lustre: server umount lustre-OST0000 complete [ 1294.830325] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1298.118539] Key type lgssc unregistered [ 1298.512899] LNet: 43880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.518070] LNetError: 43880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.545053] LNet: Removed LNI 192.168.203.159@tcp [ 1299.613326] Key type .llcrypt unregistered [ 1299.615097] Key type ._llcrypt unregistered [ 1324.066450] Key type ._llcrypt registered [ 1324.068122] Key type .llcrypt registered [ 1324.164463] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1325.481415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.621539] alg: No test for adler32 (adler32-zlib) [ 1326.646300] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1326.978589] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1328.648600] Key type lgssc registered [ 1329.865243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.450675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1338.503136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.908244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.520285] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1350.121326] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.165473] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1354.667853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.233169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1355.521394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1361.860148] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1371.364916] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1377.960479] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.460370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.902287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.598382] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1400.478905] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.498399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1416.580785] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1420.775048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.787468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.819092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.831330] Lustre: Skipped 1 previous similar message [ 1422.769707] Lustre: server umount lustre-OST0000 complete [ 1442.278860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.288962] LustreError: Skipped 1 previous similar message [ 1442.293928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.311660] Lustre: Skipped 1 previous similar message [ 1442.322280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1442.329366] Lustre: Skipped 1 previous similar message [ 1445.352738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.858144] Lustre: server umount lustre-MDT0000 complete [ 1450.470477] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.509378] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1451.601194] LustreError: 45277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384784 with bad export cookie 10351563958155266093 [ 1451.608263] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.611266] LustreError: 45277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1458.096685] Lustre: server umount lustre-MDT0001 complete [ 1470.812392] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1473.892317] Key type lgssc unregistered [ 1474.159207] LNet: 49027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.168150] LNetError: 49027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.188900] LNet: Removed LNI 192.168.203.159@tcp [ 1475.247615] Key type .llcrypt unregistered [ 1475.252416] Key type ._llcrypt unregistered [ 1500.070315] Key type ._llcrypt registered [ 1500.074145] Key type .llcrypt registered [ 1500.211977] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1501.501408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.563105] alg: No test for adler32 (adler32-zlib) [ 1502.622605] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1502.919064] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1504.599643] Key type lgssc registered [ 1505.813814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.115289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.137716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.677673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.520573] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1525.219133] LustreError: 50443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.246806] LustreError: 50443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1529.786634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.304622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.176189] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1543.065638] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.344070] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1556.425218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1556.806985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.898779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1563.695482] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1571.095611] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1580.290100] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1583.440497] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1595.817636] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 3h48m45s [ 1597.592917] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1601.006327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.015211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.036616] Lustre: Skipped 2 previous similar messages [ 1601.059206] Lustre: Skipped 2 previous similar messages [ 1602.088200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.102878] Lustre: Skipped 2 previous similar messages [ 1604.479596] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1605.782355] Lustre: server umount lustre-MDT0000 complete [ 1608.181231] LustreError: 51438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.207323] LustreError: 51438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1609.453279] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1611.614060] LustreError: 50440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.314713] LustreError: 50443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.337094] LustreError: 50443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1616.533983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.637890] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.849362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.916377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1617.915639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1621.881377] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1625.754359] Lustre: DEBUG MARKER: Wait for df (36565) ... [ 1627.330584] Lustre: DEBUG MARKER: done [ 1631.713732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.723435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.731566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.738057] Lustre: Skipped 1 previous similar message [ 1632.225120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.502913] Lustre: server umount lustre-OST0000 complete [ 1643.489955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.491811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.515918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.533868] Lustre: Skipped 2 previous similar messages [ 1643.539279] Lustre: Skipped 3 previous similar messages [ 1647.790853] Lustre: server umount lustre-MDT0000 complete [ 1650.673181] LustreError: 50438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.709640] LustreError: 50438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1651.701657] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384984 with bad export cookie 3033941706058249048 [ 1651.716614] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.727259] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1651.999783] Lustre: server umount lustre-MDT0001 complete [ 1662.323442] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1665.588791] Key type lgssc unregistered [ 1665.952128] LNet: 55368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.958132] LNetError: 55368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.987856] LNet: Removed LNI 192.168.203.159@tcp [ 1667.110350] Key type .llcrypt unregistered [ 1667.112069] Key type ._llcrypt unregistered [ 1709.349678] Key type ._llcrypt registered [ 1709.351125] Key type .llcrypt registered [ 1709.446506] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1711.042241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.134809] alg: No test for adler32 (adler32-zlib) [ 1712.173713] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1712.446432] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1714.143840] Key type lgssc registered [ 1715.348405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1723.198996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1723.249557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.582459] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1724.620679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1724.853748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.333406] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1736.989039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.047617] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1737.070297] Lustre: 57580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1737.270525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.687108] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1750.110675] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1756.378709] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1763.719067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.952728] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1764.133818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.378646] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1771.171036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1778.581310] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.754987] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 3h51m57s [ 1788.583946] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1795.706091] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1798.167366] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1798.291874] LustreError: 57589:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1818.517578] Lustre: lustre-MDT0000: Client 2a4d3a00-26cc-4f7c-a2d2-d90951a2b644 (at 192.168.203.59@tcp) reconnecting [ 1838.359126] LustreError: 57589:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1840.892562] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1843.803509] Lustre: DEBUG MARKER: done [ 1852.896424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1852.914250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.945714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.334981] Lustre: server umount lustre-OST0000 complete [ 1860.065507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1860.079743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.096435] Lustre: Skipped 1 previous similar message [ 1860.103435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.109208] Lustre: Skipped 2 previous similar messages [ 1861.154336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.162699] Lustre: Skipped 2 previous similar messages [ 1864.227291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.234588] Lustre: Skipped 1 previous similar message [ 1870.317667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.327749] Lustre: Skipped 1 previous similar message [ 1874.399317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1874.695074] Lustre: server umount lustre-MDT0000 complete [ 1878.405925] LustreError: 57574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385211 with bad export cookie 9580494474915583260 [ 1878.412966] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.422602] LustreError: 57574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1878.822656] Lustre: server umount lustre-MDT0001 complete [ 1893.429843] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1897.039716] Key type lgssc unregistered [ 1897.411842] LNet: 61913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.419120] LNetError: 61913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.442193] LNet: Removed LNI 192.168.203.159@tcp [ 1898.617641] Key type .llcrypt unregistered [ 1898.622508] Key type ._llcrypt unregistered [ 1948.855124] Key type ._llcrypt registered [ 1948.862908] Key type .llcrypt registered [ 1949.172222] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1950.653413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.764287] alg: No test for adler32 (adler32-zlib) [ 1952.023394] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1952.563066] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1954.415707] Key type lgssc registered [ 1956.069671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.296961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1960.329681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.598748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1961.615723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1961.698423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.493323] Lustre: Failing over lustre-MDT0000 [ 1962.795700] Lustre: server umount lustre-MDT0000 complete [ 1971.341351] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:55:01 (1787385301) [ 1980.047234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.842123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.713326] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1997.080587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.235759] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1997.289709] Lustre: 64709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1997.656644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.931878] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2011.151621] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2017.805361] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.061471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.300171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2025.583909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.678629] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2033.286232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2041.313692] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2047.838316] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2059.232385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2059.237577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.251344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.280622] Lustre: Skipped 1 previous similar message [ 2061.128453] Lustre: server umount lustre-MDT0000 complete [ 2065.208292] LustreError: 64702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385398 with bad export cookie 1456076363302394966 [ 2065.212376] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.218431] LustreError: 64702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2065.749445] Lustre: server umount lustre-MDT0001 complete [ 2069.586221] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2074.793966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2084.080558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2085.855957] Lustre: 63632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385402/real 1787385402] req@0000000093140c22 x1874209312647168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.892578] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.914040] Lustre: Skipped 1 previous similar message [ 2086.879193] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385403/real 1787385403] req@000000008c382828 x1874209312647296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.975170] Lustre: 63634:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385407/real 1787385407] req@00000000f0f1b0a5 x1874209312647424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.040824] LustreError: 63631:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f4de7afd x1874209312648320/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.497517] LustreError: 66561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.522526] LustreError: 66561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2091.622223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.772095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2095.648253] LustreError: 66562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.656200] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2096.671187] Lustre: 63633:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385413/real 1787385413] req@0000000075d26f3d x1874209312647808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.677944] LustreError: 68484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.710074] Lustre: 63633:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2096.745202] LustreError: 68484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2101.733097] LustreError: 66561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.754073] LustreError: 66561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2102.751107] Lustre: 63634:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385418/real 1787385418] req@00000000f39f0405 x1874209312647936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2106.040303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.470442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.002624] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2112.417656] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2115.118777] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2121.357914] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2127.709819] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2135.374692] Lustre: server umount lustre-MDT0000 complete [ 2135.526894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.533138] LustreError: 68483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.534437] Lustre: Skipped 1 previous similar message [ 2135.566984] LustreError: 68483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2139.579238] LustreError: 66567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385472 with bad export cookie 1456076363302397241 [ 2139.600636] LustreError: 66567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2139.605707] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.983561] Lustre: server umount lustre-MDT0001 complete [ 2146.860861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2156.959096] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385473/real 1787385473] req@00000000553d41b9 x1874209312674560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.997291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.007857] Lustre: Skipped 1 previous similar message [ 2157.502691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.192149] LustreError: 63631:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000084f36f72 x1874209312675712/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2164.439794] LustreError: 66561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.505575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.577059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2168.876719] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2175.976554] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2178.165320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.670955] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2183.807433] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2192.373865] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2197.849243] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2204.138986] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.144774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.165281] Lustre: Skipped 1 previous similar message [ 2204.186692] Lustre: Skipped 4 previous similar messages [ 2204.392840] Lustre: server umount lustre-MDT0000 complete [ 2209.027478] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385541 with bad export cookie 1456076363302398620 [ 2209.039390] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.041638] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2209.249926] LustreError: 71168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.264168] LustreError: 71168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2209.476883] Lustre: server umount lustre-MDT0001 complete [ 2215.728186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2220.171495] Lustre: DEBUG MARKER: files compared the same [ 2222.947403] Lustre: server umount lustre-OST0000 complete [ 2241.067665] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2244.542262] Key type lgssc unregistered [ 2244.898156] LNet: 74439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.910686] LNetError: 74439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.939193] LNet: Removed LNI 192.168.203.159@tcp [ 2245.731871] Key type .llcrypt unregistered [ 2245.738807] Key type ._llcrypt unregistered [ 2275.247531] Key type ._llcrypt registered [ 2275.250242] Key type .llcrypt registered [ 2275.388618] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2276.560112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.599789] alg: No test for adler32 (adler32-zlib) [ 2277.657782] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2277.991702] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2279.759175] Key type lgssc registered [ 2281.014413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2288.995304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.020443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.618369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.014695] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 2300.901083] LustreError: 75857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.910394] LustreError: 75857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2306.019953] LustreError: 75858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.586806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.094541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.807957] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 2321.481730] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2327.221923] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2334.689506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.919796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.981608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2340.882559] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 2349.262050] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2361.824276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.824756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.832156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.843832] Lustre: Skipped 1 previous similar message [ 2366.726500] Lustre: server umount lustre-OST0000 complete [ 2372.068424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.075279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.093253] Lustre: Skipped 1 previous similar message [ 2372.095797] Lustre: Skipped 1 previous similar message [ 2377.184660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.200486] Lustre: Skipped 2 previous similar messages [ 2377.641834] Lustre: server umount lustre-MDT0000 complete [ 2381.284966] LustreError: 75853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.305617] LustreError: 75853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2381.752848] LustreError: 77717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385714 with bad export cookie 5105573393317789830 [ 2381.765829] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.783193] LustreError: 77717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2382.236708] Lustre: server umount lustre-MDT0001 complete [ 2395.629507] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2399.397695] Key type lgssc unregistered [ 2399.832275] LNet: 79408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.849497] LNetError: 79408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.871485] LNet: Removed LNI 192.168.203.159@tcp [ 2401.044621] Key type .llcrypt unregistered [ 2401.047338] Key type ._llcrypt unregistered [ 2426.499190] Key type ._llcrypt registered [ 2426.508506] Key type .llcrypt registered [ 2426.692465] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2427.816474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.012115] alg: No test for adler32 (adler32-zlib) [ 2429.113355] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2429.428125] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2431.127194] Key type lgssc registered [ 2432.311192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.690227] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2439.709126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.327426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2460.447347] LustreError: 80800:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 2487.136501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.156611] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2501.428634] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2509.032253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.130346] Lustre: *** cfs_fail_loc=706, val=0*** [ 2509.452906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.525902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2514.251873] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2514.919772] LustreError: 80825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.949574] LustreError: 80825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2520.056419] LustreError: 80826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.086185] LustreError: 81888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.111581] LustreError: 81888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2523.611788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.872977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2528.896495] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2536.512772] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2542.596358] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2545.641705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2545.650709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.663132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.734749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.738605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.757898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.940670] Lustre: server umount lustre-OST0000 complete [ 2554.851324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.852164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.866482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.879442] Lustre: Skipped 1 previous similar message [ 2554.890273] Lustre: Skipped 2 previous similar messages [ 2559.969663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.978761] Lustre: Skipped 1 previous similar message [ 2561.053349] Lustre: server umount lustre-MDT0000 complete [ 2564.396977] LustreError: 80807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385897 with bad export cookie 372686866653275643 [ 2564.409151] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.709222] Lustre: server umount lustre-MDT0001 complete [ 2575.526361] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2578.545621] Key type lgssc unregistered [ 2578.825876] LNet: 84414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.840801] LNetError: 84414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.872452] LNet: Removed LNI 192.168.203.159@tcp [ 2579.798688] Key type .llcrypt unregistered [ 2579.802058] Key type ._llcrypt unregistered [ 2608.651234] Key type ._llcrypt registered [ 2608.659032] Key type .llcrypt registered [ 2608.806769] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2610.013116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.029952] alg: No test for adler32 (adler32-zlib) [ 2611.071353] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2611.370545] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2613.055203] Key type lgssc registered [ 2614.096364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.085714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.105313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.391644] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2635.698595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.496431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2641.542679] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2641.893349] LustreError: 86517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.923416] LustreError: 86517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2647.010290] LustreError: 86516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.032127] LustreError: 86516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2650.993083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2651.325036] LustreError: 86515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.363957] LustreError: 86515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2651.387589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.747450] LustreError: 86543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.780702] LustreError: 86543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2658.637851] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2661.860799] LustreError: 86520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.895933] LustreError: 86520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2667.527785] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.542927] Lustre: MGS: Client 36664748-0e1b-40a4-8a9b-a26bfac14eb1 (at 0@lo) reconnecting [ 2667.562847] LustreError: 86531:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2667.814454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.903182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2672.595754] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2688.488461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.504359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.519881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2693.604366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2693.618995] Lustre: Skipped 2 previous similar messages [ 2694.081756] Lustre: server umount lustre-OST0000 complete [ 2698.720822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.721388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.729372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.741326] Lustre: Skipped 2 previous similar messages [ 2703.844915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.851739] Lustre: Skipped 2 previous similar messages [ 2704.244782] Lustre: server umount lustre-MDT0000 complete [ 2708.984782] LustreError: 86521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.030711] LustreError: 86521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2709.659250] Lustre: server umount lustre-MDT0001 complete [ 2714.421408] Lustre: server umount MGS complete [ 2727.729182] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:07:38 (1787386058) [ 2760.784299] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2771.426320] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2809.278356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.872347] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2825.088617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.217648] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2825.248425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2825.294955] Lustre: lustre-MDT0001: new disk, initializing [ 2825.350345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.370388] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2825.378029] Lustre: 93713:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2825.383906] LustreError: 93713:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2829.209888] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2840.887054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2841.144886] Lustre: lustre-OST0000: new disk, initializing [ 2841.153869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2841.163474] Lustre: 94479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2841.231837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.384159] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2848.799804] Lustre: 94853:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001f401c01 x1874210003657088/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787386197 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2857.521901] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2865.339996] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.350398] Lustre: MGS: Client f119e339-77d3-44a0-84d5-1fcbf80411a8 (at 0@lo) reconnecting [ 2865.436755] Lustre: 95525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2865.456961] LustreError: 93698:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2865.473770] LustreError: 93698:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2865.514904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2865.685289] Lustre: lustre-MDT0000: new disk, initializing [ 2865.924624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.972355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2866.011128] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2867.040945] Lustre: 95549:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000935a89fe x1874210003664000/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787386215 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2870.622734] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2872.225634] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2876.202138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2876.210576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2876.286467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2901.986816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2901.992524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.997525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2902.004375] LustreError: Skipped 1 previous similar message [ 2902.020779] Lustre: Skipped 1 previous similar message [ 2914.275022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.285816] Lustre: Skipped 5 previous similar messages [ 2915.295106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.534203] Lustre: server umount lustre-OST0000 complete [ 2919.396329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.404112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.405816] Lustre: Skipped 2 previous similar messages [ 2925.250494] Lustre: server umount lustre-MDT0000 complete [ 2935.062699] Lustre: server umount lustre-MDT0001 complete [ 2939.692167] Lustre: server umount MGS complete [ 2954.899520] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:11:25 (1787386285) [ 2973.364356] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2976.141510] Key type lgssc unregistered [ 2976.397854] LNet: 98051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.404210] LNetError: 98051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2977.449055] LNet: Removed LNI 192.168.203.159@tcp [ 2978.336699] Key type .llcrypt unregistered [ 2978.339139] Key type ._llcrypt unregistered [ 2995.504369] Key type ._llcrypt registered [ 2995.509252] Key type .llcrypt registered [ 2995.633936] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2996.480721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2996.540857] alg: No test for adler32 (adler32-zlib) [ 2997.764524] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 2998.026570] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2999.696070] Key type lgssc registered [ 3000.851466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3008.057050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3008.088429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.635695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.053527] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3019.252252] LustreError: 99274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.289496] LustreError: 99274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3022.275455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.656924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3027.206736] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3034.620250] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.191706] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3046.737731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.108805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.217695] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3055.363895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3059.977500] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3070.951964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3070.953623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.995280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3071.006207] Lustre: Skipped 1 previous similar message [ 3071.023443] Lustre: Skipped 1 previous similar message [ 3075.514776] Lustre: server umount lustre-OST0000 complete [ 3079.137268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.155598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.170397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.184748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.187896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.509390] Lustre: server umount lustre-MDT0000 complete [ 3086.317662] LustreError: 99275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.149963] LustreError: 99255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386422 with bad export cookie 2141387846123911188 [ 3089.155071] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.163106] LustreError: 99255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3089.576351] Lustre: server umount lustre-MDT0001 complete [ 3104.259431] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3107.425159] Key type lgssc unregistered [ 3107.800822] LNet: 102808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3107.808188] LNetError: 102808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3107.828229] LNet: Removed LNI 192.168.203.159@tcp [ 3108.661422] Key type .llcrypt unregistered [ 3108.664410] Key type ._llcrypt unregistered [ 3123.670429] Key type ._llcrypt registered [ 3123.673224] Key type .llcrypt registered [ 3123.772403] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3124.987926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3125.014266] alg: No test for adler32 (adler32-zlib) [ 3126.047309] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3126.275880] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3127.935163] Key type lgssc registered [ 3129.125342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.472829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.484115] LustreError: 103886:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3135.510371] LustreError: 103893:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3135.515139] LustreError: 103886:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3135.517975] LustreError: 103886:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3135.522050] LustreError: 103886:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3135.531892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.102866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.017445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.334043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3150.300222] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3156.046126] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing lsmod [ 3159.009278] LustreError: 105090:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3159.051796] LustreError: 105099:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3159.057839] LustreError: 105090:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 3159.065988] LustreError: 105090:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3159.075460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.077674] LustreError: 105090:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3159.494888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.537382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3166.192749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3166.209706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.247583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.786754] Lustre: server umount lustre-MDT0000 complete [ 3170.787478] LustreError: 103931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.804772] LustreError: 103931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3172.275994] LustreError: 103908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386505 with bad export cookie 9817778367633257495 [ 3172.284112] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.290464] LustreError: 103908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3172.871293] LustreError: 104362:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002db753f4 x1874210543653632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3173.024996] Lustre: server umount lustre-MDT0001 complete [ 3186.802748] Lustre: server umount lustre-OST0000 complete [ 3195.119604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.585224] LustreError: 106162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.661791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.002603] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3200.992788] LustreError: 106163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.115849] LustreError: 106162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.054137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.457753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3213.044260] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3220.558306] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3225.985693] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3232.365092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.696728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.710863] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3241.917873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3245.637582] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.243373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.248615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.262932] Lustre: Skipped 2 previous similar messages [ 3254.275252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.280446] Lustre: Skipped 2 previous similar messages [ 3257.314182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3257.317706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.322247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.346842] Lustre: Skipped 1 previous similar message [ 3258.847378] Lustre: server umount lustre-OST0000 complete [ 3265.008735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3265.019517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.050466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3268.876831] Lustre: server umount lustre-MDT0000 complete [ 3272.683013] LustreError: 106162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.710275] LustreError: 106162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3272.777365] LustreError: 106145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386605 with bad export cookie 9817778367633258426 [ 3272.778530] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.798138] LustreError: 106145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3273.065190] Lustre: server umount lustre-MDT0001 complete [ 3285.114738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3288.023358] Key type lgssc unregistered [ 3288.377384] LNet: 109697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.386259] LNetError: 109697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.409562] LNet: Removed LNI 192.168.203.159@tcp [ 3289.328448] Key type .llcrypt unregistered [ 3289.331663] Key type ._llcrypt unregistered [ 3313.342869] Key type ._llcrypt registered [ 3313.346836] Key type .llcrypt registered [ 3313.535760] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3315.085707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3315.118042] alg: No test for adler32 (adler32-zlib) [ 3316.321184] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3316.609842] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3318.319591] Key type lgssc registered [ 3319.778851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3328.109489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3328.133934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.645304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.092360] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3339.237196] LustreError: 111113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.263384] LustreError: 111113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3342.546535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.786318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3347.479885] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3355.214488] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3361.649361] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3369.768445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.104316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.234681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3377.074365] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3385.970701] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3394.164989] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3397.909087] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3403.241322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.245302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.270169] Lustre: Skipped 1 previous similar message [ 3408.353074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.365567] Lustre: Skipped 2 previous similar messages [ 3409.216404] Lustre: server umount lustre-OST0000 complete [ 3413.473366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.492622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.502896] Lustre: Skipped 1 previous similar message [ 3418.593690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.602507] Lustre: Skipped 1 previous similar message [ 3419.740649] Lustre: server umount lustre-MDT0000 complete [ 3423.583155] LustreError: 111094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386756 with bad export cookie 14701431667149546425 [ 3423.587564] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3423.593924] LustreError: 111094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3424.094487] Lustre: server umount lustre-MDT0001 complete [ 3436.228812] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3439.287458] Key type lgssc unregistered [ 3439.661427] LNet: 114753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3439.678415] LNetError: 114753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3439.738061] LNet: Removed LNI 192.168.203.159@tcp [ 3440.791758] Key type .llcrypt unregistered [ 3440.795430] Key type ._llcrypt unregistered [ 3460.644274] Key type ._llcrypt registered [ 3460.645969] Key type .llcrypt registered [ 3460.890867] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3462.821300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3462.899526] alg: No test for adler32 (adler32-zlib) [ 3464.153320] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3464.574854] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3466.399292] Key type lgssc registered [ 3468.790295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3478.737509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3478.780842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.532028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.649427] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3490.787106] LustreError: 115976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.794117] LustreError: 115976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3494.231101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.511345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3498.698130] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3507.731543] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3514.649039] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3522.345452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.931121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.005156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3529.505179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3538.969618] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3550.693269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.696728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3550.709313] Lustre: Skipped 1 previous similar message [ 3550.747431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.048507] Lustre: server umount lustre-OST0000 complete [ 3560.928944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.930489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3560.950415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.970922] Lustre: Skipped 2 previous similar messages [ 3564.044739] Lustre: server umount lustre-MDT0000 complete [ 3566.054086] LustreError: 115977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.080959] LustreError: 115977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3568.460467] LustreError: 115957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386901 with bad export cookie 15106621919114696941 [ 3568.471360] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.474122] LustreError: 115957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3568.940706] Lustre: server umount lustre-MDT0001 complete [ 3584.075067] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3587.536660] Key type lgssc unregistered [ 3587.917792] LNet: 119509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3587.937546] LNetError: 119509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.965864] LNet: Removed LNI 192.168.203.159@tcp [ 3588.856659] Key type .llcrypt unregistered [ 3588.860188] Key type ._llcrypt unregistered [ 3617.206466] Key type ._llcrypt registered [ 3617.208677] Key type .llcrypt registered [ 3617.323855] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3618.328595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3618.417206] alg: No test for adler32 (adler32-zlib) [ 3619.389340] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3619.561986] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3621.195772] Key type lgssc registered [ 3622.598410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.837508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3630.860861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.534885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.843233] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3642.848238] LustreError: 120928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.871582] LustreError: 120928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3647.250899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.698718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.533695] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3663.519995] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3670.730518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3678.995061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.330689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.918909] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3694.204690] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3702.905016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3711.767947] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3715.252757] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3715.262787] Lustre: Skipped 2 previous similar messages [ 3719.252910] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3719.273954] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3719.937499] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3719.947773] Lustre: Skipped 2 previous similar messages [ 3726.406241] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3726.416113] Lustre: Skipped 1 previous similar message [ 3732.613992] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3732.636020] Lustre: Skipped 2 previous similar messages [ 3736.364750] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.59@tcp 192.168.0.[2,10]@tcp [ 3741.485150] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.59/24@tcp in log lustre-MDT0000 [ 3741.493865] Lustre: Skipped 2 previous similar messages [ 3746.402065] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.59/24@tcp [ 3746.415170] Lustre: Skipped 1 previous similar message [ 3757.033201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3757.036602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.047867] LustreError: Skipped 1 previous similar message [ 3757.072386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.575741] Lustre: server umount lustre-OST0000 complete [ 3764.710355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.713748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.733612] Lustre: Skipped 1 previous similar message [ 3769.085334] Lustre: server umount lustre-MDT0000 complete [ 3769.829736] LustreError: 120924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.858649] LustreError: 120924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3772.962422] LustreError: 120907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387105 with bad export cookie 851791021663025291 [ 3772.972738] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.974744] LustreError: 120907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3782.459412] Lustre: server umount lustre-MDT0001 complete [ 3794.047167] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3797.510554] Key type lgssc unregistered [ 3797.780898] LNet: 125902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3797.785422] LNetError: 125902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3797.802115] LNet: Removed LNI 192.168.203.159@tcp [ 3798.406341] Key type .llcrypt unregistered [ 3798.409658] Key type ._llcrypt unregistered [ 3831.177514] Key type ._llcrypt registered [ 3831.182426] Key type .llcrypt registered [ 3831.282396] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3832.737893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3832.811461] alg: No test for adler32 (adler32-zlib) [ 3833.922323] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3834.185279] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3835.872528] Key type lgssc registered [ 3837.161653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3846.097580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3846.119750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.630131] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3847.645608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.665513] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.59/24@tcp [ 3853.201635] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3857.898330] LustreError: 127469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.951140] LustreError: 127469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3863.013315] LustreError: 127468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.327117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.735808] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3863.763990] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.59/24@tcp [ 3863.822805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3869.776438] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3878.759248] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3885.147855] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3892.582439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.142573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.590531] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3910.467527] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3917.782521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3933.159171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.162219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.171626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.207652] Lustre: Skipped 1 previous similar message [ 3934.507363] Lustre: server umount lustre-OST0000 complete [ 3940.838643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3940.844821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.877947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.888473] Lustre: Skipped 1 previous similar message [ 3943.393258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.393874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.418738] Lustre: Skipped 1 previous similar message [ 3945.151446] Lustre: server umount lustre-MDT0000 complete [ 3948.530127] LustreError: 127469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.562725] LustreError: 127469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3949.062168] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387281 with bad export cookie 11135450488184010634 [ 3949.077501] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3949.088030] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3949.412388] Lustre: server umount lustre-MDT0001 complete [ 3964.759560] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3967.764157] Key type lgssc unregistered [ 3968.276912] LNet: 131082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3968.292388] LNetError: 131082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3968.317369] LNet: Removed LNI 192.168.203.159@tcp [ 3969.407691] Key type .llcrypt unregistered [ 3969.409440] Key type ._llcrypt unregistered [ 4006.817326] Key type ._llcrypt registered [ 4006.821314] Key type .llcrypt registered [ 4007.072387] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4025.626570] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4026.913093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4026.939560] alg: No test for adler32 (adler32-zlib) [ 4028.033452] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4028.247793] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4030.000581] Key type lgssc registered [ 4031.358616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.513398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.544289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.884994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4075.952529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4076.050980] Lustre: lustre-MDT0000: new disk, initializing [ 4076.151750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.174924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4081.806525] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4093.738882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.946676] Lustre: 134487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4094.016816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4094.031475] Lustre: Skipped 1 previous similar message [ 4094.176634] Lustre: lustre-MDT0001: new disk, initializing [ 4094.305520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.356436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4094.368964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4100.821635] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4112.645880] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4119.653559] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4127.881086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4128.156792] Lustre: lustre-OST0000: new disk, initializing [ 4128.160484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4128.166957] Lustre: 136456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4128.250598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.538370] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4136.515775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4136.540863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4142.074457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4146.811119] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4152.258895] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4152.525777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4158.116977] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4158.399789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4162.535326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.543840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.548962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.556724] LustreError: Skipped 1 previous similar message [ 4167.652823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.668210] Lustre: Skipped 1 previous similar message [ 4172.772467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.783708] Lustre: Skipped 2 previous similar messages [ 4175.327243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4175.747693] Lustre: server umount lustre-OST0000 complete [ 4181.475801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4181.486670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.502458] Lustre: Skipped 1 previous similar message [ 4181.510622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.008676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.490318] Lustre: server umount lustre-MDT0000 complete [ 4191.246624] LustreError: 134480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387524 with bad export cookie 846846784983036732 [ 4191.263793] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.269509] LustreError: 134480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4191.682177] Lustre: server umount lustre-MDT0001 complete [ 4201.688509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.343602] LustreError: 138676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.375920] LustreError: 138676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4202.457915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.591520] LustreError: 138677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.051355] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4212.711186] LustreError: 138676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.826766] LustreError: 138677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.020479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.582558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4224.343987] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4232.966464] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4239.614570] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4247.177138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.459109] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4247.472716] Lustre: Skipped 2 previous similar messages [ 4247.616226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.998440] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4265.677162] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4269.984309] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4270.211156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4274.448801] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4274.696702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.486501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.680861] Lustre: lustre-OST0001: new disk, initializing [ 4282.684890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4282.690387] Lustre: 142050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4282.781308] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4289.588548] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4291.664993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4291.673852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4291.772091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4298.617402] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4298.962961] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4306.169258] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4306.552511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4315.358962] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cee02d2f800.ost_server_uuid 50 [ 4318.122957] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cee02d2f800.ost_server_uuid in IDLE state after 0 sec [ 4327.921062] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4327.922413] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.947525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4327.949050] Lustre: Skipped 1 previous similar message [ 4327.971720] Lustre: Skipped 1 previous similar message [ 4333.929753] Lustre: server umount lustre-OST0001 complete [ 4338.154214] LustreError: 140501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4338.191309] LustreError: 140501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4339.680889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.701698] LustreError: Skipped 1 previous similar message [ 4339.705782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4344.524901] Lustre: server umount lustre-OST0000 complete [ 4351.457347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.469813] LustreError: Skipped 1 previous similar message [ 4351.471789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.499271] Lustre: Skipped 1 previous similar message [ 4351.510382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4351.515670] Lustre: Skipped 6 previous similar messages [ 4354.949895] Lustre: server umount lustre-MDT0000 complete [ 4358.626603] LustreError: 140934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.658526] LustreError: 140934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4358.678273] LustreError: 139411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387691 with bad export cookie 846846784983037614 [ 4358.682218] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.693935] LustreError: 139411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4358.981915] Lustre: server umount lustre-MDT0001 complete [ 4380.995476] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4384.363320] Key type lgssc unregistered [ 4384.839922] LNet: 145056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4384.843476] LNetError: 145056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4385.893782] LNet: Removed LNI 192.168.203.159@tcp [ 4387.171654] Key type .llcrypt unregistered [ 4387.178772] Key type ._llcrypt unregistered [ 4434.240163] Key type ._llcrypt registered [ 4434.246734] Key type .llcrypt registered [ 4434.378766] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4448.819639] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4449.904685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.965930] alg: No test for adler32 (adler32-zlib) [ 4451.063615] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4451.200322] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4452.881349] Key type lgssc registered [ 4454.231129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4494.666511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4494.749805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.061776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.131471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4496.266384] Lustre: lustre-MDT0000: new disk, initializing [ 4496.376367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.395688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4500.121754] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4512.365573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.508883] Lustre: 149223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.545224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4512.547697] Lustre: Skipped 1 previous similar message [ 4512.641261] Lustre: lustre-MDT0001: new disk, initializing [ 4512.776570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4512.854846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4512.875476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.927303] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4527.241380] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4533.108745] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4540.046511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.377257] Lustre: lustre-OST0000: new disk, initializing [ 4540.381237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4540.391788] Lustre: 151194:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4540.523555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.182924] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4548.156683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4548.167186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4548.295559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4556.719554] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4561.170547] Lustre: Failing over lustre-OST0000 [ 4561.533898] Lustre: server umount lustre-OST0000 complete [ 4578.279221] Lustre: 147450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387895/real 1787387895] req@000000008e22dd87 x1874211933029376/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787387911 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4578.317864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.295621] Lustre: 147448:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387896/real 1787387896] req@00000000e2c09a40 x1874211933029632/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787387912 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4579.386143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.489304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.788344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.816537] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4584.857167] Lustre: 147450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387901/real 1787387901] req@0000000069e65262 x1874211933031424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4584.879106] Lustre: 147450:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4588.975970] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4589.843442] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4590.491422] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4590.493395] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4591.519410] Lustre: 147450:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387907/real 1787387907] req@000000001e914dd2 x1874211933032576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4593.607910] Lustre: Failing over lustre-MDT0000 [ 4593.984362] Lustre: server umount lustre-MDT0000 complete [ 4594.081831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.099752] Lustre: Skipped 1 previous similar message [ 4599.267851] LustreError: 149232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.303657] LustreError: 149232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4600.671408] LustreError: 151201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.386831] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.404934] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4609.504448] Lustre: 147449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387926/real 1787387926] req@0000000023682f44 x1874211933044736/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787387942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4609.516804] LustreError: 149236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.539493] Lustre: 147449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4609.540285] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.598473] LustreError: 149236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4613.471328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.627620] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.660482] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4619.746946] LustreError: 147447:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000027f09bf9 x1874211933050240/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4620.030601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.105896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4622.174544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4624.192919] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4625.395625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4625.399573] Lustre: Skipped 1 previous similar message [ 4625.430371] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4625.471661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4635.620456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.631980] Lustre: Skipped 1 previous similar message [ 4635.639353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.739787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.745335] Lustre: Skipped 2 previous similar messages [ 4645.874902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4645.880889] Lustre: Skipped 1 previous similar message [ 4646.367742] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4646.707576] Lustre: server umount lustre-OST0000 complete [ 4650.989571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.999767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4651.007732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4651.013316] Lustre: Skipped 2 previous similar messages [ 4656.097490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4656.102431] Lustre: Skipped 2 previous similar messages [ 4665.311113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4665.595222] Lustre: server umount lustre-MDT0000 complete [ 4666.336544] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.361379] LustreError: 150402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4669.456124] LustreError: 149217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388002 with bad export cookie 4771525227090447021 [ 4669.461837] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.471920] LustreError: 149217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4669.829347] Lustre: server umount lustre-MDT0001 complete [ 4683.092699] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4686.670793] Key type lgssc unregistered [ 4687.025572] LNet: 154851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4687.030524] LNetError: 154851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4687.066643] LNet: Removed LNI 192.168.203.159@tcp [ 4688.200597] Key type .llcrypt unregistered [ 4688.207366] Key type ._llcrypt unregistered [ 4727.187140] Key type ._llcrypt registered [ 4727.188902] Key type .llcrypt registered [ 4727.299371] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4729.014413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4729.066889] alg: No test for adler32 (adler32-zlib) [ 4730.258536] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4730.648303] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4732.408467] Key type lgssc registered [ 4733.635291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4741.875713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4741.902898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.410851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.649093] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4753.894871] LustreError: 156375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.928276] LustreError: 156375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4757.618658] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.017810] LustreError: 156374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.704234] LustreError: 156403:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4761.710149] LustreError: 156403:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4761.713606] LustreError: 156403:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4761.972161] Lustre: server umount lustre-MDT0000 complete [ 4778.136811] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4788.279340] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4825.526386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.739767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4825.775857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4825.852289] Lustre: lustre-MDT0000: new disk, initializing [ 4825.909257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.929319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4830.487932] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4840.693968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.826780] Lustre: 159707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4840.889315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4840.892611] Lustre: Skipped 1 previous similar message [ 4840.946962] Lustre: lustre-MDT0001: new disk, initializing [ 4841.013705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4841.048161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4841.060964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4845.617378] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4857.012374] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4864.250371] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.953823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4873.131374] Lustre: lustre-OST0000: new disk, initializing [ 4873.134947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4873.147227] Lustre: 161685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4873.232546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.592704] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4880.467076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4880.485431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4880.658768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4889.174204] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4894.017555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4895.562428] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:43:46 (1787388226) [ 5157.343956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.343956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.344428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.358065] Lustre: Skipped 1 previous similar message [ 5157.373635] Lustre: Skipped 1 previous similar message [ 5162.463760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5162.468490] Lustre: Skipped 3 previous similar messages [ 5167.584304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5167.601417] Lustre: Skipped 2 previous similar messages [ 5171.167359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5171.372942] Lustre: server umount lustre-MDT0000 complete [ 5172.716484] LustreError: 163291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.748749] LustreError: 163291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5175.526989] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388508 with bad export cookie 8800028556529873707 [ 5175.530848] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.541112] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5177.823802] LustreError: 159714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.825608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.851626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5181.875589] Lustre: server umount lustre-MDT0001 complete [ 5199.839125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5200.067282] Lustre: server umount lustre-OST0000 complete [ 5210.292592] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 5223.709604] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5260.441559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.740799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5260.767158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5260.853707] Lustre: lustre-MDT0000: new disk, initializing [ 5260.994505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.018215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5265.579590] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5275.705939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.848076] Lustre: 166348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5275.902345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5275.907739] Lustre: Skipped 1 previous similar message [ 5276.006147] Lustre: lustre-MDT0001: new disk, initializing [ 5276.111213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5276.170144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5276.189423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5280.656517] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5288.979434] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5294.210821] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.813591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.135177] Lustre: lustre-OST0000: new disk, initializing [ 5301.139311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5301.147884] Lustre: 168320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5301.230151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.499565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5307.511427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5307.723249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5307.738874] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5318.587776] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5323.049738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5323.275782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5327.735624] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5328.000710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5333.472395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.476566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.484233] LustreError: Skipped 1 previous similar message [ 5333.488329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5333.496559] Lustre: Skipped 1 previous similar message [ 5333.503158] Lustre: Skipped 1 previous similar message [ 5336.098409] Lustre: server umount lustre-OST0000 complete [ 5342.691421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5342.702139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.724210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.726996] Lustre: Skipped 1 previous similar message [ 5346.114649] Lustre: server umount lustre-MDT0000 complete [ 5348.846746] LustreError: 166357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.877509] LustreError: 166357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5350.040386] LustreError: 168319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388682 with bad export cookie 8800028556530032537 [ 5350.042114] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.051356] LustreError: 168319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5350.534705] Lustre: server umount lustre-MDT0001 complete [ 5359.006964] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:51:29 (1787388689) [ 5374.880249] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 5383.332865] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5419.859864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.029728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5420.064365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5420.154173] Lustre: lustre-MDT0000: new disk, initializing [ 5420.229579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.246597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5424.986486] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5434.194808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.284862] Lustre: 173076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5434.529707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5439.032285] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5447.715196] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5452.569268] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5459.356649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5459.672375] Lustre: lustre-OST0000: new disk, initializing [ 5459.675645] Lustre: Skipped 1 previous similar message [ 5459.682310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5459.690927] Lustre: Skipped 2 previous similar messages [ 5459.698543] Lustre: 175055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5465.687859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5465.703644] Lustre: Skipped 1 previous similar message [ 5465.711383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5465.866403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5466.147917] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5475.344112] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5486.562674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.565440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5486.576300] Lustre: Skipped 1 previous similar message [ 5486.594106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.597023] Lustre: Skipped 2 previous similar messages [ 5488.892706] Lustre: server umount lustre-OST0000 complete [ 5495.778889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5495.795475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.808791] Lustre: Skipped 1 previous similar message [ 5499.106219] Lustre: server umount lustre-MDT0000 complete [ 5501.923927] LustreError: 173083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5501.945211] LustreError: 173083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5502.852758] LustreError: 173069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388835 with bad export cookie 8800028556530033419 [ 5502.859599] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.869629] LustreError: 173069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5503.116668] Lustre: server umount lustre-MDT0001 complete [ 5511.527079] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:54:02 (1787388842) [ 5526.428670] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 5534.923420] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5566.728562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.978805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5567.036136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5567.182258] Lustre: lustre-MDT0000: new disk, initializing [ 5567.284175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.296405] Lustre: Skipped 2 previous similar messages [ 5567.313715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5572.073589] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5583.048669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.235331] Lustre: 179375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5583.683981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5588.715907] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5598.447436] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5603.834486] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5610.616684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.959456] Lustre: 181352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5617.440303] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5620.824321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5620.947582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5626.824634] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5636.582356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5636.583091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.599384] LustreError: Skipped 1 previous similar message [ 5636.604435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5636.615160] Lustre: Skipped 2 previous similar messages [ 5636.637861] Lustre: Skipped 4 previous similar messages [ 5641.092289] Lustre: server umount lustre-OST0000 complete [ 5651.839736] Lustre: server umount lustre-MDT0000 complete [ 5651.945540] LustreError: 179387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.960936] LustreError: 179387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5656.369560] LustreError: 179368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388989 with bad export cookie 8800028556530034504 [ 5656.380410] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5656.383908] LustreError: 179368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5656.779366] Lustre: server umount lustre-MDT0001 complete [ 5669.768405] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5672.754598] Key type lgssc unregistered [ 5673.112561] LNet: 183208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5673.118298] LNetError: 183208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5673.148577] LNet: Removed LNI 192.168.203.159@tcp [ 5673.910980] Key type .llcrypt unregistered [ 5673.914190] Key type ._llcrypt unregistered [ 5696.387243] Key type ._llcrypt registered [ 5696.391183] Key type .llcrypt registered [ 5696.497995] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5697.229566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5697.266191] alg: No test for adler32 (adler32-zlib) [ 5698.356213] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 5698.623973] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5700.335398] Key type lgssc registered [ 5701.569264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5708.731191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5708.745258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.108947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.242440] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5719.529392] LustreError: 184626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.545112] LustreError: 184626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5722.151078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.610177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5726.145986] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5732.964788] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5738.472126] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5744.816725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.194854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.376286] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5754.422390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5758.921049] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5777.888625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5777.902423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.924294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.936369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.938560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5779.953614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.629597] Lustre: server umount lustre-OST0000 complete [ 5789.154502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5789.167217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.183493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5789.187084] Lustre: Skipped 1 previous similar message [ 5792.890160] Lustre: server umount lustre-MDT0000 complete [ 5795.309445] LustreError: 184621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.332148] LustreError: 184621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5797.173094] LustreError: 186467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389130 with bad export cookie 16501925741358474534 [ 5797.179489] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.183679] LustreError: 186467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5797.620417] Lustre: server umount lustre-MDT0001 complete [ 5808.837219] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5811.670422] Key type lgssc unregistered [ 5812.023839] LNet: 188180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5812.033852] LNetError: 188180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5812.050648] LNet: Removed LNI 192.168.203.159@tcp [ 5813.110072] Key type .llcrypt unregistered [ 5813.114512] Key type ._llcrypt unregistered [ 5839.415458] Key type ._llcrypt registered [ 5839.418426] Key type .llcrypt registered [ 5839.559990] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5840.838135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5840.878444] alg: No test for adler32 (adler32-zlib) [ 5842.119949] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 5842.367575] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5844.048889] Key type lgssc registered [ 5845.120448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5853.972521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5853.999990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.544792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.912792] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5865.957859] LustreError: 189597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.976709] LustreError: 189597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5869.796537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.237531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5875.069948] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5881.731587] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5887.171590] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5893.498851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.870347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.735143] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5901.041955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5906.111160] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5913.984480] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 5916.641216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.645068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.646088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.653504] Lustre: Skipped 1 previous similar message [ 5916.682858] Lustre: Skipped 1 previous similar message [ 5919.076231] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 5921.767638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.783153] Lustre: Skipped 1 previous similar message [ 5926.369234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5926.526378] Lustre: server umount lustre-OST0000 complete [ 5933.015822] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9cee09075800.ost_server_uuid 50 [ 5934.730477] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5951.460768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.466702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.475506] Lustre: Skipped 1 previous similar message [ 5951.481473] Lustre: Skipped 1 previous similar message [ 5954.890601] Lustre: server umount lustre-MDT0000 complete [ 5956.577341] LustreError: 189597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.601472] LustreError: 189597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5958.470503] LustreError: 189580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389291 with bad export cookie 4132284566454117524 [ 5958.478082] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.493396] LustreError: 189580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5958.807850] Lustre: server umount lustre-MDT0001 complete [ 5965.670203] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:01:36 (1787389296) [ 5971.064385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.428166] LustreError: 193525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.549883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.698590] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5976.547472] LustreError: 193526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.488379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.667452] LustreError: 193525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.868922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5985.855617] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5992.583767] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5997.740502] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6003.547034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.874255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.386351] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6016.182247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6016.467635] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6023.322425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.541505] Lustre: lustre-OST0001: new disk, initializing [ 6023.547090] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6023.557191] Lustre: 196366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6023.647113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.574669] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6033.042954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6033.049825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6033.151741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6038.644977] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6042.086487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.096969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.115935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.121378] Lustre: Skipped 1 previous similar message [ 6043.105163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6047.647882] Lustre: server umount lustre-OST0000 complete [ 6048.741418] LustreError: 195348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.760339] LustreError: 195348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6053.402080] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6053.660895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6057.727171] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6057.969994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6064.992297] LustreError: 195348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.013620] LustreError: 195348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6074.347397] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.348213] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.355470] LustreError: Skipped 1 previous similar message [ 6074.357588] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6074.368300] Lustre: Skipped 2 previous similar messages [ 6074.384547] Lustre: Skipped 4 previous similar messages [ 6077.088945] Lustre: server umount lustre-OST0001 complete [ 6084.592388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6084.592621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.616600] Lustre: Skipped 1 previous similar message [ 6087.861431] Lustre: server umount lustre-MDT0000 complete [ 6091.250926] LustreError: 196365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389424 with bad export cookie 4132284566454118595 [ 6091.257944] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.264203] LustreError: 196365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.624116] Lustre: server umount lustre-MDT0001 complete [ 6114.085298] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:04:05 (1787389445) [ 6119.937969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.203211] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6120.264732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6120.475280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.599665] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6132.080917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.170781] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6132.209615] Lustre: 199805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6136.813480] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6143.074979] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6148.607660] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6155.573835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6155.824097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6162.039650] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6162.564452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6169.031396] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6176.043547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6176.109908] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6182.547129] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6184.143342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6191.048322] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6194.661774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.675446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.685841] Lustre: Skipped 3 previous similar messages [ 6200.563951] Lustre: server umount lustre-OST0000 complete [ 6202.215213] LustreError: 201658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.241628] LustreError: 201658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6219.746208] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6219.754751] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.763118] Lustre: Skipped 1 previous similar message [ 6222.732372] Lustre: server umount lustre-OST0001 complete [ 6232.759653] Lustre: server umount lustre-MDT0000 complete [ 6237.062263] LustreError: 199799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389569 with bad export cookie 4132284566454120002 [ 6237.065772] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.076101] LustreError: 199799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.466547] Lustre: server umount lustre-MDT0001 complete [ 6261.770142] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:06:32 (1787389592) [ 6276.959068] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 6287.174567] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6323.574480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.776771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6323.831112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6323.947731] Lustre: lustre-MDT0000: new disk, initializing [ 6324.033808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.038978] Lustre: Skipped 3 previous similar messages [ 6324.048984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6328.165924] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6337.653671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.757443] Lustre: 207904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6337.811922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6337.821763] Lustre: Skipped 1 previous similar message [ 6337.948363] Lustre: lustre-MDT0001: new disk, initializing [ 6338.111583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6338.120916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6341.788564] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6349.566297] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6355.448873] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6362.305223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.586251] Lustre: lustre-OST0000: new disk, initializing [ 6362.593277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6362.604157] Lustre: 209872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6369.853979] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6369.898148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6369.912049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6370.051590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6378.418908] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6381.963980] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6382.102056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6385.509273] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6385.905744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6390.754298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6390.759497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.761397] LustreError: Skipped 2 previous similar messages [ 6390.764610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6390.764618] Lustre: Skipped 9 previous similar messages [ 6390.799891] Lustre: Skipped 4 previous similar messages [ 6394.179255] Lustre: server umount lustre-OST0000 complete [ 6411.231126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6411.346405] Lustre: server umount lustre-MDT0000 complete [ 6414.203330] LustreError: 207896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389747 with bad export cookie 4132284566454121787 [ 6414.204725] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.211159] LustreError: 207896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.515175] Lustre: server umount lustre-MDT0001 complete [ 6421.514990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.955990] LustreError: 212088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.978884] LustreError: 212088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6426.287153] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6434.055826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.464915] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6445.224625] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6450.771881] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6456.008941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.342519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.348045] Lustre: Skipped 4 previous similar messages [ 6460.991461] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6467.181236] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6471.122305] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6471.311594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6474.849946] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6475.050344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6476.779749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.786355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6476.795763] Lustre: Skipped 3 previous similar messages [ 6476.816099] LustreError: Skipped 2 previous similar messages [ 6482.623688] Lustre: server umount lustre-OST0000 complete [ 6487.248183] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6491.126818] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6520.924474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.585383] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6531.756306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.770048] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.773273] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6531.802123] Lustre: Skipped 1 previous similar message [ 6532.671201] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6537.375480] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6542.315431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.316301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.322761] Lustre: Skipped 13 previous similar messages [ 6542.336316] LustreError: Skipped 1 previous similar message [ 6546.450161] Lustre: server umount lustre-OST0000 complete [ 6564.319203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6564.463380] Lustre: server umount lustre-MDT0000 complete [ 6567.473227] LustreError: 212069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389900 with bad export cookie 4132284566454122669 [ 6567.478073] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.487328] LustreError: 212069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6567.765681] Lustre: server umount lustre-MDT0001 complete [ 6575.415569] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:11:46 (1787389906) [ 6580.896020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.369687] LustreError: 218055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.392354] LustreError: 218055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6585.253067] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6592.452628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.707771] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6604.042655] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6609.991945] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6617.403242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.001462] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6631.132278] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6636.247114] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6636.485905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6641.016665] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6641.235770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6647.321563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.511378] Lustre: lustre-OST0001: new disk, initializing [ 6647.513954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6647.518874] Lustre: 221424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6653.513492] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6657.156468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6657.186670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6657.311533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6664.161278] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6669.256311] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6669.540346] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6674.243130] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6674.467651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6677.987891] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.995397] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.008260] Lustre: Skipped 5 previous similar messages [ 6678.016513] LustreError: Skipped 1 previous similar message [ 6682.352632] Lustre: server umount lustre-OST0001 complete [ 6688.219564] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6688.431570] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6692.757902] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6692.977216] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6721.433429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6721.637771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6721.642883] Lustre: Skipped 5 previous similar messages [ 6726.645559] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6726.669747] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6726.670278] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6727.316197] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6734.756188] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6736.246096] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6752.223137] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6752.308862] Lustre: server umount lustre-OST0001 complete [ 6773.732910] Lustre: server umount lustre-OST0000 complete [ 6783.432100] Lustre: server umount lustre-MDT0000 complete [ 6786.687609] LustreError: 218036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390119 with bad export cookie 4132284566454123635 [ 6786.691735] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.707716] LustreError: 218036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6793.157941] Lustre: server umount lustre-MDT0001 complete [ 6814.521503] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:15:45 (1787390145) [ 6820.319969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.524223] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6820.561228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6824.360792] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6831.691420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.814667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6831.838319] Lustre: 226664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.090560] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6842.585399] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6847.684761] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6853.165555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6853.295074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6859.033269] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6865.685690] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6873.737428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.811868] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6873.890088] LustreError: 228510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.915342] LustreError: 228510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 6879.467250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6885.729182] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6887.147462] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6890.828402] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6891.021306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6895.107895] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6895.329302] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6901.004144] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cee18d92000.ost_server_uuid 50 [ 6902.436591] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cee18d92000.ost_server_uuid in IDLE state after 0 sec [ 6903.244931] Lustre: Permanently deactivating lustre-OST0001 [ 6903.263829] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6906.579828] Lustre: Permanently reactivating lustre-OST0001 [ 6909.921729] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6909.925783] Lustre: Skipped 25 previous similar messages [ 6916.018386] Lustre: server umount lustre-OST0001 complete [ 6925.131803] Lustre: server umount lustre-OST0000 complete [ 6934.492422] Lustre: server umount lustre-MDT0000 complete [ 6937.832851] LustreError: 226657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390270 with bad export cookie 4132284566454124923 [ 6937.839136] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6937.843856] LustreError: 226657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6938.287550] Lustre: server umount lustre-MDT0001 complete [ 6960.219174] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:18:11 (1787390291) [ 6967.534277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.861090] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6967.904379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6967.917955] Lustre: Skipped 5 previous similar messages [ 6971.856391] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6978.253492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.318094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6978.336200] Lustre: 233355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6982.471857] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6989.982700] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6995.502705] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7001.322659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.474120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7001.519301] Lustre: Permanently deactivating lustre-OST0000 [ 7001.537060] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7006.647558] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7010.825309] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7013.280481] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7019.400404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.480269] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7020.483778] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7024.487387] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7030.947162] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7032.590435] Lustre: Permanently reactivating lustre-OST0000 [ 7032.609991] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7032.620949] Lustre: Skipped 2 previous similar messages [ 7041.073841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.083829] Lustre: Skipped 12 previous similar messages [ 7041.088858] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7041.095279] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7041.100979] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7041.105233] Lustre: Skipped 1 previous similar message [ 7041.132416] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7041.140839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7045.609726] Lustre: server umount lustre-OST0001 complete [ 7046.115000] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.134018] LustreError: Skipped 9 previous similar messages [ 7055.498437] Lustre: server umount lustre-OST0000 complete [ 7065.756620] Lustre: server umount lustre-MDT0000 complete [ 7069.355327] LustreError: 236234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390402 with bad export cookie 4132284566454126561 [ 7069.363879] LustreError: 236234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7069.364869] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7069.718154] Lustre: server umount lustre-MDT0001 complete [ 7081.664456] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7084.136635] Key type lgssc unregistered [ 7084.417688] LNet: 238213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.433391] LNetError: 238213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.450127] LNet: Removed LNI 192.168.203.159@tcp [ 7085.315826] Key type .llcrypt unregistered [ 7085.321985] Key type ._llcrypt unregistered [ 7105.440601] Key type ._llcrypt registered [ 7105.443654] Key type .llcrypt registered [ 7105.527639] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7106.234311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.371063] alg: No test for adler32 (adler32-zlib) [ 7107.346908] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7107.597204] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7109.242661] Key type lgssc registered [ 7110.177906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7117.668500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7117.693288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.098900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.426542] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7128.546547] LustreError: 239678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.563792] LustreError: 239678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7129.031914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.198584] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7129.231424] Lustre: 239668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7129.240388] Lustre: Permanently deactivating lustre-MDT0001 [ 7129.250055] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7129.442656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7132.489830] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7132.498240] LustreError: 239709:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7132.515948] LustreError: 239709:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7132.527874] LustreError: 239709:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7132.733603] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7139.412930] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7144.280528] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7150.750145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.096895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.152635] LustreError: 241517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.430459] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7159.271364] LustreError: 241519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.363508] LustreError: 241519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.374331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7163.396343] LustreError: 241519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7164.105711] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7168.486661] LustreError: 242136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.510755] LustreError: 242136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7170.351857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.582744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.691870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7175.705585] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7176.445967] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7184.348494] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7188.192938] Lustre: Permanently reactivating lustre-MDT0001 [ 7188.231439] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7188.242901] Lustre: Skipped 2 previous similar messages [ 7191.945492] Lustre: lustre-MDT0001: Client e9c8aa76-0001-4fff-83bc-674463a16403 (at 192.168.203.59@tcp) reconnecting [ 7199.252937] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7201.901840] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7207.739851] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7208.060442] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7214.291762] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7214.539779] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7218.049689] Lustre: Permanently deactivating lustre-MDT0001 [ 7218.088342] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7218.095747] Lustre: Skipped 2 previous similar messages [ 7224.945133] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7231.970684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7231.982373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.994336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.019769] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.021499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.038294] Lustre: Skipped 2 previous similar messages [ 7234.049985] Lustre: Skipped 3 previous similar messages [ 7234.220314] Lustre: server umount lustre-MDT0000 complete [ 7237.859518] LustreError: 239658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390570 with bad export cookie 1916549201352293261 [ 7237.865350] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7237.881395] LustreError: 239658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7238.225697] Lustre: server umount lustre-MDT0001 complete [ 7242.003590] Lustre: server umount lustre-OST0000 complete [ 7253.062499] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:23:03 (1787390583) [ 7261.967191] Lustre: server umount lustre-OST0001 complete [ 7268.729410] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 7278.359624] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7312.370801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.721404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7312.727369] Lustre: Skipped 2 previous similar messages [ 7312.756620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7312.826199] Lustre: lustre-MDT0000: new disk, initializing [ 7312.957092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.979308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7318.236938] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7328.757606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.880389] Lustre: 247935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7328.950685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7328.954583] Lustre: Skipped 1 previous similar message [ 7329.114469] Lustre: lustre-MDT0001: new disk, initializing [ 7329.252227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.300032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7329.326059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7333.911485] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7343.080420] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7348.943780] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7356.507238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7356.807347] Lustre: lustre-OST0000: new disk, initializing [ 7356.815328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7356.821046] Lustre: 249904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7356.906276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.675033] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7366.737989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7366.757638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7366.893657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.016875] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7376.173466] LustreError: 250624:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7384.546987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.719420] Lustre: lustre-OST0001: new disk, initializing [ 7384.725213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7384.733578] Lustre: 251021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7386.271540] LustreError: 250624:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7386.302808] LustreError: 249543:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7391.344734] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7393.896584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7393.912219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7394.043295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7396.392802] LustreError: 249543:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7400.709713] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7402.373198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7404.521891] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7404.522588] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.547507] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7404.550969] Lustre: Skipped 1 previous similar message [ 7409.635537] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7409.639233] Lustre: Skipped 1 previous similar message [ 7414.760554] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7414.771441] Lustre: Skipped 1 previous similar message [ 7418.335391] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7418.420170] Lustre: server umount lustre-OST0001 complete [ 7419.875741] LustreError: 250370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.888232] LustreError: 250370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7435.247646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.266364] Lustre: Skipped 1 previous similar message [ 7435.280967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7435.291352] Lustre: Skipped 1 previous similar message [ 7440.332567] Lustre: server umount lustre-OST0000 complete [ 7445.474357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.487664] Lustre: Skipped 2 previous similar messages [ 7450.966438] Lustre: server umount lustre-MDT0000 complete [ 7455.733594] LustreError: 247946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.765090] LustreError: 247946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7455.885256] LustreError: 247926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390788 with bad export cookie 1916549201352295858 [ 7455.892319] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.905720] LustreError: 247926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.404284] Lustre: server umount lustre-MDT0001 complete [ 7473.544924] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7476.349847] Key type lgssc unregistered [ 7476.597643] LNet: 253054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7476.610761] LNetError: 253054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7476.627053] LNet: Removed LNI 192.168.203.159@tcp [ 7477.488943] Key type .llcrypt unregistered [ 7477.490755] Key type ._llcrypt unregistered [ 7518.608858] Key type ._llcrypt registered [ 7518.611062] Key type .llcrypt registered [ 7518.760529] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7520.272959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7520.308775] alg: No test for adler32 (adler32-zlib) [ 7521.468571] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7521.981854] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7523.679293] Key type lgssc registered [ 7525.087732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7533.180506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7533.202212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.542204] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7534.557207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7534.768133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7538.779750] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7546.019044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.217714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7546.271633] Lustre: 255260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7546.537546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.017413] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7557.321706] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7563.015052] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7569.523771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.716837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7569.909366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.715147] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7576.752272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7582.709206] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7637.985661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7637.992950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.034557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7638.496232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7638.514858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.534906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7638.539540] Lustre: Skipped 1 previous similar message [ 7640.204537] Lustre: server umount lustre-OST0000 complete [ 7644.230391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7657.014291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.369167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7658.413467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7658.428779] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7663.530575] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7663.559722] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7663.777928] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7672.137914] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7714.783738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7714.789831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.795664] LustreError: Skipped 1 previous similar message [ 7714.802834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7714.817851] Lustre: Skipped 1 previous similar message [ 7714.841257] Lustre: Skipped 1 previous similar message [ 7718.244435] Lustre: server umount lustre-OST0000 complete [ 7725.024318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.027309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7725.039536] Lustre: Skipped 1 previous similar message [ 7725.068319] Lustre: Skipped 1 previous similar message [ 7727.961710] Lustre: server umount lustre-MDT0000 complete [ 7731.024080] LustreError: 255254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391063 with bad export cookie 11477334093220830775 [ 7731.025792] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7731.037763] LustreError: 255254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7731.455476] Lustre: server umount lustre-MDT0001 complete [ 7743.395433] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7747.455660] Key type lgssc unregistered [ 7747.902671] LNet: 260683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7747.937732] LNetError: 260683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7747.983293] LNet: Removed LNI 192.168.203.159@tcp [ 7749.055773] Key type .llcrypt unregistered [ 7749.058274] Key type ._llcrypt unregistered [ 7782.465285] Key type ._llcrypt registered [ 7782.467793] Key type .llcrypt registered [ 7782.547208] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7783.517445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7783.527218] alg: No test for adler32 (adler32-zlib) [ 7784.897518] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7785.281943] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7786.943752] Key type lgssc registered [ 7788.108565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7795.355508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7795.386762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.879875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.704568] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7806.434360] LustreError: 262102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.451256] LustreError: 262102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7808.746813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.233547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7812.959207] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7820.355753] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7825.962970] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7833.105287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.652539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.952922] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7841.856591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7847.706991] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7870.439837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7870.444293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.468837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7870.475917] Lustre: Skipped 1 previous similar message [ 7875.618223] Lustre: server umount lustre-OST0000 complete [ 7880.674842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.678966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.696435] Lustre: Skipped 2 previous similar messages [ 7885.437352] Lustre: server umount lustre-MDT0000 complete [ 7885.796789] LustreError: 263168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.812362] LustreError: 263168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7888.630198] LustreError: 262083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391221 with bad export cookie 15793113581261730812 [ 7888.634903] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.643169] LustreError: 262083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7888.995682] Lustre: server umount lustre-MDT0001 complete [ 7899.367292] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7902.542729] Key type lgssc unregistered [ 7902.730961] LNet: 266405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7902.752904] LNetError: 266405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7902.814631] LNet: Removed LNI 192.168.203.159@tcp [ 7904.446225] Key type .llcrypt unregistered [ 7904.460438] Key type ._llcrypt unregistered [ 7924.610673] Key type ._llcrypt registered [ 7924.612794] Key type .llcrypt registered [ 7924.720505] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7925.970725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7926.040046] alg: No test for adler32 (adler32-zlib) [ 7927.066340] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7927.304862] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7929.008851] Key type lgssc registered [ 7930.068463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7936.920097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7936.938587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.240192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.593388] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7948.773709] LustreError: 267627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.791808] LustreError: 267627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7949.377288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.611719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7953.957501] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7960.884391] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7966.476481] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7972.980497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.464706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.291226] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7983.756318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7987.387330] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7999.457266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.465910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.494279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.502830] Lustre: Skipped 1 previous similar message [ 8003.731524] Lustre: server umount lustre-OST0000 complete [ 8009.699358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.702161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.708866] Lustre: Skipped 2 previous similar messages [ 8009.717751] Lustre: Skipped 1 previous similar message [ 8013.106648] Lustre: server umount lustre-MDT0000 complete [ 8014.820413] LustreError: 267627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.831545] LustreError: 267627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8016.221908] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391349 with bad export cookie 108185950033803149 [ 8016.237946] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.486093] Lustre: server umount lustre-MDT0001 complete [ 8027.652412] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8030.345903] Key type lgssc unregistered [ 8030.625408] LNet: 271305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8030.637631] LNetError: 271305:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8030.650665] LNet: Removed LNI 192.168.203.159@tcp [ 8031.384138] Key type .llcrypt unregistered [ 8031.388061] Key type ._llcrypt unregistered [ 8053.509583] Key type ._llcrypt registered [ 8053.513362] Key type .llcrypt registered [ 8053.623179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8054.728194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8054.739656] alg: No test for adler32 (adler32-zlib) [ 8055.772259] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8055.965633] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8057.607163] Key type lgssc registered [ 8058.760693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.678967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.698739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.117182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.210274] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8077.287749] LustreError: 272722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.309099] LustreError: 272722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8080.917178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8081.493106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8086.113567] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8094.254288] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8099.645619] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8107.117808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.645975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.766996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8115.597252] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8123.848874] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8147.939111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.942238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.946652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.967524] Lustre: Skipped 1 previous similar message [ 8153.066737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8153.079812] Lustre: Skipped 2 previous similar messages [ 8153.698258] Lustre: server umount lustre-OST0000 complete [ 8158.183826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.188575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8158.198727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8163.173839] Lustre: server umount lustre-MDT0000 complete [ 8163.303715] LustreError: 272723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.322601] LustreError: 272723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8166.453444] LustreError: 272702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391499 with bad export cookie 6402478005897849961 [ 8166.467888] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8166.471084] LustreError: 272702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8166.706977] Lustre: server umount lustre-MDT0001 complete [ 8177.181510] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8180.444644] Key type lgssc unregistered [ 8180.857288] LNet: 277031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8180.865213] LNetError: 277031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8181.927516] LNet: Removed LNI 192.168.203.159@tcp [ 8182.943056] Key type .llcrypt unregistered [ 8182.944453] Key type ._llcrypt unregistered [ 8202.709763] Key type ._llcrypt registered [ 8202.712541] Key type .llcrypt registered [ 8202.854675] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8203.689880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8203.709593] alg: No test for adler32 (adler32-zlib) [ 8204.812504] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8205.120817] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8206.887159] Key type lgssc registered [ 8207.976615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8216.078211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.089364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.689798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.315903] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8227.815585] LustreError: 278269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8227.836024] LustreError: 278269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8230.441926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.783085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.194989] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8242.261584] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8247.617092] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8253.723502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8254.148454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.713472] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8262.415623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8267.409869] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8282.090395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.101482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.115662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8283.105486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8283.114267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8283.117301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.133171] Lustre: Skipped 1 previous similar message [ 8284.694825] Lustre: server umount lustre-OST0000 complete [ 8288.226438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.232608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.240404] Lustre: Skipped 1 previous similar message [ 8288.259325] Lustre: Skipped 1 previous similar message [ 8293.353565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.368213] Lustre: Skipped 1 previous similar message [ 8294.495632] Lustre: server umount lustre-MDT0000 complete [ 8298.476398] LustreError: 278260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.500281] LustreError: 278260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8299.021801] LustreError: 278236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391631 with bad export cookie 16361815283740989442 [ 8299.048357] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.426788] Lustre: server umount lustre-MDT0001 complete [ 8311.902458] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8314.952603] Key type lgssc unregistered [ 8315.515745] LNet: 281970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8315.521194] LNetError: 281970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.534674] LNet: Removed LNI 192.168.203.159@tcp [ 8316.533564] Key type .llcrypt unregistered [ 8316.535419] Key type ._llcrypt unregistered [ 8330.138123] Key type ._llcrypt registered [ 8330.145204] Key type .llcrypt registered [ 8330.309079] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8347.649715] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 8363.571940] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8365.521176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8365.581450] alg: No test for adler32 (adler32-zlib) [ 8366.759720] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8367.146116] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8368.887350] Key type lgssc registered [ 8370.209639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.470768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.492726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.810045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8406.860714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8406.934460] Lustre: lustre-MDT0000: new disk, initializing [ 8407.020338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.038713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8412.052946] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8426.473192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.656382] Lustre: 285529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8426.773917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8426.791905] Lustre: Skipped 1 previous similar message [ 8426.870617] Lustre: lustre-MDT0001: new disk, initializing [ 8427.021869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.058932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8427.085807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8433.009267] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8443.285446] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8448.155478] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8456.072065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.487582] Lustre: lustre-OST0000: new disk, initializing [ 8456.495679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8456.503128] Lustre: 287498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8456.626225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.986367] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8465.477152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.500583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8465.611657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8471.704660] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8477.365248] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8477.735219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8482.899108] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8483.159815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8486.376114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8486.379425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.396361] LustreError: Skipped 1 previous similar message [ 8486.397915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.430624] Lustre: Skipped 1 previous similar message [ 8491.175948] Lustre: server umount lustre-OST0000 complete [ 8496.609527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.625931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.630704] Lustre: Skipped 1 previous similar message [ 8496.645723] Lustre: Skipped 2 previous similar messages [ 8501.739533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.769135] Lustre: Skipped 1 previous similar message [ 8501.907434] Lustre: server umount lustre-MDT0000 complete [ 8505.901250] LustreError: 286325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391838 with bad export cookie 18066730758353151073 [ 8505.907217] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8505.941648] LustreError: 286325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8506.503387] Lustre: server umount lustre-MDT0001 complete [ 8518.553936] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:44:08 (1787391848) [ 8525.570892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.118452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8530.312975] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8531.430851] LustreError: 289909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.444576] LustreError: 289909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8535.522991] LustreError: 289910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.708187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.162086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8543.624682] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8552.418818] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8558.267182] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8565.066731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8565.457812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8570.910628] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8577.854080] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8594.403180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.456789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8598.469276] Lustre: server umount lustre-OST0000 complete [ 8604.642300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.665049] Lustre: Skipped 1 previous similar message [ 8604.671789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.691341] Lustre: Skipped 2 previous similar messages [ 8608.788308] Lustre: server umount lustre-MDT0000 complete [ 8609.772181] LustreError: 289910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.785257] LustreError: 289910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8612.283398] LustreError: 289890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391945 with bad export cookie 18066730758353151955 [ 8612.285085] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8612.291272] LustreError: 289890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8612.586686] Lustre: server umount lustre-MDT0001 complete [ 8623.946226] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8627.075855] Key type lgssc unregistered [ 8627.392734] LNet: 293466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8627.400859] LNetError: 293466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8627.413355] LNet: Removed LNI 192.168.203.159@tcp [ 8628.496498] Key type .llcrypt unregistered [ 8628.498538] Key type ._llcrypt unregistered [ 8660.731329] Key type ._llcrypt registered [ 8660.734671] Key type .llcrypt registered [ 8660.988446] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 8677.076716] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8678.106284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8678.160311] alg: No test for adler32 (adler32-zlib) [ 8679.218969] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8679.434069] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8681.111193] Key type lgssc registered [ 8682.264823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8727.589130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8727.625745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.944244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8728.985669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8729.083537] Lustre: lustre-MDT0000: new disk, initializing [ 8729.173094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.196114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8734.096960] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8743.564315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.667646] Lustre: 297293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8743.720975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8743.724169] Lustre: Skipped 1 previous similar message [ 8743.822260] Lustre: lustre-MDT0001: new disk, initializing [ 8743.918571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8743.953571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8743.968495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8748.614749] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8758.434030] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8764.291981] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8771.226960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8771.428505] Lustre: lustre-OST2710: new disk, initializing [ 8771.436944] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8771.444574] Lustre: 299263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8771.567338] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8777.228503] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8778.354428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8778.382692] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8778.502067] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8787.345518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8796.045513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8796.283434] Lustre: lustre-OST03e8: new disk, initializing [ 8796.291476] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8796.298431] Lustre: 300325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8796.456734] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8803.039717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8803.077186] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8803.122250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8803.308522] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8811.013031] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8819.195548] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8819.444880] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8823.898472] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8824.156130] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8827.753727] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8827.911359] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8831.929468] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8832.132885] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8836.065049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8836.071466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.082059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.140711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.142230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.170128] Lustre: Skipped 1 previous similar message [ 8839.173051] Lustre: Skipped 1 previous similar message [ 8842.331919] Lustre: server umount lustre-MDT0000 complete [ 8845.537551] LustreError: 297288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392178 with bad export cookie 15408672163211561905 [ 8845.546345] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8845.553316] LustreError: 297288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8845.906748] Lustre: server umount lustre-MDT0001 complete [ 8859.708860] Lustre: server umount lustre-OST2710 complete [ 8872.922166] Lustre: server umount lustre-OST03e8 complete [ 8888.178373] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 8897.469982] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8931.410791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.841410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8931.934209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8932.164542] Lustre: lustre-MDT0000: new disk, initializing [ 8932.339202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.394412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8937.399770] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8947.094175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.266555] Lustre: 305578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8947.351033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8947.362069] Lustre: Skipped 1 previous similar message [ 8947.474620] Lustre: lustre-MDT0001: new disk, initializing [ 8947.642042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8947.715300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8947.748389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8952.895331] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8963.153103] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8969.486728] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8976.617636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8976.871903] Lustre: lustre-OST0000: new disk, initializing [ 8976.875968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8976.889763] Lustre: 307543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8976.980239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8983.204373] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8986.155701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8986.163970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8986.242360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8991.404823] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8995.142380] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8995.361907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8999.368429] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8999.636725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9001.444461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.452679] Lustre: Skipped 1 previous similar message [ 9001.457561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9001.463023] Lustre: Skipped 3 previous similar messages [ 9006.565605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9006.578695] Lustre: Skipped 1 previous similar message [ 9007.333852] Lustre: server umount lustre-OST0000 complete [ 9011.688880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.694623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9011.702656] Lustre: Skipped 2 previous similar messages [ 9011.717341] Lustre: Skipped 1 previous similar message [ 9017.185941] Lustre: server umount lustre-MDT0000 complete [ 9020.761192] LustreError: 305569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392353 with bad export cookie 15408672163211563585 [ 9020.769670] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9020.776311] LustreError: 305569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9021.111702] Lustre: server umount lustre-MDT0001 complete [ 9028.774466] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:52:39 (1787392359) [ 9030.274698] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9032.054280] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:52:42 (1787392362) [ 9035.698309] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9060.909730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.136436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9061.164788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9061.307296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.506084] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9073.149104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.315900] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9073.349240] Lustre: 311403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9078.262421] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9099.829757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9100.089913] Lustre: Denying initial registration attempt from nid 192.168.203.159@tcp, specified as failover [ 9100.119753] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9100.141184] LustreError: 312854:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9100.154156] LustreError: 312854:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9100.160647] LustreError: 312854:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9100.260519] Lustre: server umount lustre-OST0000 complete [ 9100.265200] LustreError: 312854:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9109.477038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9109.494891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9109.513438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9109.524516] Lustre: Skipped 2 previous similar messages [ 9112.908043] Lustre: server umount lustre-MDT0000 complete [ 9116.132217] LustreError: 311414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9116.175366] LustreError: 311414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9117.030181] LustreError: 311394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392449 with bad export cookie 15408672163211564467 [ 9117.032290] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9117.047497] LustreError: 311394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9117.432620] Lustre: server umount lustre-MDT0001 complete [ 9128.162472] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9130.744817] Key type lgssc unregistered [ 9131.013647] LNet: 313976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9131.021356] LNetError: 313976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9131.044172] LNet: Removed LNI 192.168.203.159@tcp [ 9131.898256] Key type .llcrypt unregistered [ 9131.902831] Key type ._llcrypt unregistered [ 9142.118717] Key type ._llcrypt registered [ 9142.121842] Key type .llcrypt registered [ 9142.239882] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9143.096088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9143.115388] alg: No test for adler32 (adler32-zlib) [ 9144.103709] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9144.388533] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9146.103823] Key type lgssc registered [ 9147.546251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9195.488430] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9202.658809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9202.673668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.919866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9203.940826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9204.070302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.986480] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9215.851910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.953654] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9215.984342] Lustre: 316730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.222717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9219.482487] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9226.412883] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9232.836314] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9252.408807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9252.641881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9252.901299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.359575] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9267.142898] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9269.216872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9269.222674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9269.237425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9269.750666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9269.758899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9269.778435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.851126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.869801] Lustre: Skipped 1 previous similar message [ 9275.444274] Lustre: server umount lustre-OST0000 complete [ 9279.973616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.981109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9280.004617] Lustre: Skipped 1 previous similar message [ 9280.027538] Lustre: Skipped 1 previous similar message [ 9285.092491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9285.104873] Lustre: Skipped 1 previous similar message [ 9285.828863] Lustre: server umount lustre-MDT0000 complete [ 9289.622548] LustreError: 316723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392622 with bad export cookie 8313713252901834498 [ 9289.628284] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9289.640643] LustreError: 316723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9289.935395] Lustre: server umount lustre-MDT0001 complete [ 9301.060179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9303.806215] Key type lgssc unregistered [ 9304.102688] LNet: 320301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9304.118039] LNetError: 320301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9304.139654] LNet: Removed LNI 192.168.203.159@tcp [ 9304.959131] Key type .llcrypt unregistered [ 9304.961057] Key type ._llcrypt unregistered [ 9326.735340] Key type ._llcrypt registered [ 9326.736830] Key type .llcrypt registered [ 9326.829349] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9327.698489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9327.722386] alg: No test for adler32 (adler32-zlib) [ 9328.731079] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9328.989066] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9330.665407] Key type lgssc registered [ 9331.729181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9338.996845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9339.025485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.582163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.957065] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9350.118408] LustreError: 321718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.139590] LustreError: 321718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9353.198603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.564437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.126312] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9364.596138] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9369.221635] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9377.069082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9377.397158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.843764] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9389.522309] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9399.776388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9399.782076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9399.799189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.314772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.318709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.348740] Lustre: Skipped 2 previous similar messages [ 9403.703234] Lustre: server umount lustre-MDT0000 complete [ 9406.434189] LustreError: 321718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.455489] LustreError: 321718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9407.259686] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392740 with bad export cookie 16402244062199288395 [ 9407.269797] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9407.273627] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9407.363147] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 9407.368718] Lustre: Skipped 1 previous similar message [ 9411.555146] LustreError: 321715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.555534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.577876] LustreError: 321715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9411.604913] Lustre: Skipped 1 previous similar message [ 9411.610483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9413.649042] Lustre: server umount lustre-MDT0001 complete [ 9420.657250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9430.362056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.268558] LustreError: 323552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.293848] LustreError: 323552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9434.351639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9434.439577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9439.018920] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9442.273832] LustreError: 325310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9442.295927] LustreError: 325310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9442.536631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9447.484646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.892438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9452.601735] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9453.035338] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9460.196986] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9461.781476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9467.415362] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9468.915516] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9475.552735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9475.567739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.599168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9475.604630] Lustre: Skipped 2 previous similar messages [ 9478.625501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9483.747046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9483.756692] Lustre: Skipped 3 previous similar messages [ 9488.351574] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9488.521569] Lustre: server umount lustre-OST0000 complete [ 9493.985913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9493.986943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.009842] Lustre: Skipped 2 previous similar messages [ 9498.575471] Lustre: server umount lustre-MDT0000 complete [ 9499.113458] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9499.144343] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9502.105961] LustreError: 326051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392835 with bad export cookie 16402244062199296144 [ 9502.121284] LustreError: 326051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9502.124807] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9502.427646] Lustre: server umount lustre-MDT0001 complete [ 9513.356861] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9516.213981] Key type lgssc unregistered [ 9516.513370] LNet: 327987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9516.534472] LNetError: 327987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9516.556192] LNet: Removed LNI 192.168.203.159@tcp [ 9517.389443] Key type .llcrypt unregistered [ 9517.393072] Key type ._llcrypt unregistered [ 9543.284698] Key type ._llcrypt registered [ 9543.286975] Key type .llcrypt registered [ 9543.368158] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9544.220197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9544.314629] alg: No test for adler32 (adler32-zlib) [ 9545.415873] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9545.715266] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9547.375194] Key type lgssc registered [ 9548.638683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9556.465628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9556.484720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.014089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.033974] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9567.210897] LustreError: 329417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9567.234100] LustreError: 329417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9569.803200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9570.173260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9574.460608] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9581.514420] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9586.772360] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9590.753440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9590.761226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.775733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.805520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.808241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.938690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.946626] Lustre: Skipped 2 previous similar messages [ 9600.998642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.006760] Lustre: Skipped 1 previous similar message [ 9604.063631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9604.337871] Lustre: server umount lustre-MDT0000 complete [ 9606.112778] LustreError: 330145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9606.140357] LustreError: 330145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9607.799874] LustreError: 329397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392940 with bad export cookie 18140581870118162723 [ 9607.820524] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9608.031336] Lustre: server umount lustre-MDT0001 complete [ 9621.757613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.984448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9622.002691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9622.137792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9628.391925] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9636.678237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.773623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9636.799206] Lustre: 332006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9637.053501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9641.169554] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9648.194940] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9652.971459] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9662.256407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.397776] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9662.404949] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9662.412634] LustreError: 333891:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9662.428912] LustreError: 333891:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9662.445417] LustreError: 333891:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9662.497250] Lustre: server umount lustre-OST0000 complete [ 9662.500127] LustreError: 333891:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9672.314506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.554925] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9672.783196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.158593] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9682.586362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9686.160927] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9692.490722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9692.574261] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9692.664450] Lustre: lustre-OST0001: new disk, initializing [ 9692.667425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9692.672694] Lustre: 335430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9692.753827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.388736] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9699.943486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9699.954336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9700.092348] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9706.713836] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9710.577054] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.577109] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.583615] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9710.590123] LustreError: Skipped 1 previous similar message [ 9710.617778] Lustre: Skipped 1 previous similar message [ 9720.815636] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9720.832143] Lustre: Skipped 4 previous similar messages [ 9723.359189] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9723.477629] Lustre: server umount lustre-OST0001 complete [ 9725.930610] LustreError: 334359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.945337] LustreError: 334359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9728.480041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9728.485931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.063048] LustreError: 334357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9731.098378] LustreError: 334357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9732.869673] Lustre: server umount lustre-OST0000 complete [ 9739.753829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9739.769589] LustreError: Skipped 1 previous similar message [ 9739.779483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.794497] Lustre: Skipped 1 previous similar message [ 9739.803415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9739.812031] Lustre: Skipped 4 previous similar messages [ 9743.111866] Lustre: server umount lustre-MDT0000 complete [ 9746.403469] LustreError: 332017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.432378] LustreError: 332017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9746.893259] LustreError: 332000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393079 with bad export cookie 18140581870118163269 [ 9746.902780] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9746.916954] LustreError: 332000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9747.260920] Lustre: server umount lustre-MDT0001 complete [ 9757.497943] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9760.514031] Key type lgssc unregistered [ 9760.845568] LNet: 337392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9760.860754] LNetError: 337392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9760.891336] LNet: Removed LNI 192.168.203.159@tcp [ 9761.722981] Key type .llcrypt unregistered [ 9761.725087] Key type ._llcrypt unregistered [ 9828.409242] Key type ._llcrypt registered [ 9828.413229] Key type .llcrypt registered [ 9828.501774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 9843.418714] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9844.326557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9844.416412] alg: No test for adler32 (adler32-zlib) [ 9845.385808] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9845.600119] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9847.312360] Key type lgssc registered [ 9848.707953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9879.888506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9879.905801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.214953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9881.271686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9881.382341] Lustre: lustre-MDT0000: new disk, initializing [ 9881.490924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9881.509851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9885.571170] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9895.391974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.561290] Lustre: 342324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9895.618654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9895.622359] Lustre: Skipped 1 previous similar message [ 9895.746732] Lustre: lustre-MDT0001: new disk, initializing [ 9895.891337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.930795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9895.941675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9900.659833] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9909.588253] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9914.836268] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9922.819852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9923.121236] Lustre: lustre-OST0000: new disk, initializing [ 9923.125151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9923.130938] Lustre: 344296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9923.252833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9929.986345] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9931.909575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9931.932499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9932.084534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9940.068666] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9944.213536] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9944.401350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9948.783272] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9948.971844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9952.736247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9952.749358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.754771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9952.764445] Lustre: Skipped 1 previous similar message [ 9952.787654] Lustre: Skipped 1 previous similar message [ 9957.071466] Lustre: server umount lustre-OST0000 complete [ 9962.464036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9962.483136] LustreError: Skipped 1 previous similar message [ 9962.500861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9962.533118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9966.841513] Lustre: server umount lustre-MDT0000 complete [ 9970.487060] LustreError: 343121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393303 with bad export cookie 13954597883703840980 [ 9970.492569] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.506493] LustreError: 343121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9970.865508] Lustre: server umount lustre-MDT0001 complete [ 9978.949734] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:08:29 (1787393309) [ 9986.013443] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:08:36 (1787393316) [ 9994.153953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.602348] LustreError: 347050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9994.614088] LustreError: 347050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9994.664866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9998.887462] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9999.841501] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.961031] LustreError: 347050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.791874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.332478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10012.769581] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10019.708507] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10024.565433] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10031.204526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10031.641920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10037.788789] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10044.461470] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10052.491463] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10054.021765] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10055.636949] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10057.258580] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10059.331510] Lustre: Failing over lustre-MDT0000 [10059.602894] Lustre: server umount lustre-MDT0000 complete [10060.258564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.270395] LustreError: 347045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.285193] Lustre: Skipped 3 previous similar messages [10060.323938] LustreError: 347045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10065.379381] LustreError: 347050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.403644] LustreError: 347050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10075.625495] LustreError: 347045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.641045] LustreError: 347045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10076.639230] Lustre: 340548:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393393/real 1787393393] req@000000003b3d7b8e x1874217589084416/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787393409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10076.656630] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.682600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.886632] LustreError: 340547:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fee812a3 x1874217589089920/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10087.186978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.250610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10088.299693] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10091.535810] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10092.541532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10092.573622] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10092.643994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10100.950946] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10102.387889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10105.921921] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10107.795988] Lustre: Failing over lustre-MDT0000 [10107.875810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10107.878476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.889062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.904160] Lustre: Skipped 2 previous similar messages [10107.914654] Lustre: Skipped 3 previous similar messages [10108.125225] Lustre: server umount lustre-MDT0000 complete [10108.798243] LustreError: 348334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.810188] LustreError: 348334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10126.400150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10126.508132] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10126.747266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10126.800817] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10129.253370] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10131.639722] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10131.985508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10131.998257] Lustre: Skipped 2 previous similar messages [10132.046834] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10132.093176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10141.047897] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10142.698408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10152.424334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10152.425650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.427228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10152.455560] Lustre: Skipped 2 previous similar messages [10155.955792] Lustre: server umount lustre-MDT0000 complete [10157.541326] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10157.582783] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [10159.177261] LustreError: 347031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393492 with bad export cookie 13954597883703843696 [10159.182606] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.189033] LustreError: 347031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10159.496885] Lustre: server umount lustre-MDT0001 complete [10173.775073] Lustre: server umount lustre-OST0000 complete [10187.920586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.110157] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10204.880597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10205.489613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10205.496845] Lustre: Skipped 1 previous similar message [10210.305373] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10217.074644] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10222.345029] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10228.459526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10233.844684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10234.505777] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10242.216116] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10248.941713] Lustre: server umount lustre-OST0000 complete [10254.324342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.325490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10254.334323] Lustre: Skipped 1 previous similar message [10254.362788] Lustre: Skipped 3 previous similar messages [10262.578104] Lustre: server umount lustre-MDT0000 complete [10264.545052] LustreError: 353404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.576947] LustreError: 353404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10265.631088] Lustre: 340550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393582/real 1787393582] req@00000000cd309bab x1874217589154816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787393598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10266.260136] LustreError: 353384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393599 with bad export cookie 13954597883703844445 [10266.271140] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10266.275231] LustreError: 353384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10266.535975] Lustre: server umount lustre-MDT0001 complete [10276.916968] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [10279.334932] Key type lgssc unregistered [10279.665793] LNet: 356934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10279.673505] LNetError: 356934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10279.688163] LNet: Removed LNI 192.168.203.159@tcp [10280.374389] Key type .llcrypt unregistered [10280.376568] Key type ._llcrypt unregistered [10306.663115] Key type ._llcrypt registered [10306.667442] Key type .llcrypt registered [10306.802150] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [10307.653378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10307.678726] alg: No test for adler32 (adler32-zlib) [10308.834780] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [10309.096251] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [10310.727224] Key type lgssc registered [10311.971872] Lustre: Echo OBD driver; http://www.lustre.org/ [10319.260432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10319.294618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10320.775971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10325.108468] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10331.113266] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.135204] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10332.049444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10332.406380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10335.880394] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10341.647247] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10347.445724] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10353.375690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.835220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10360.242388] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10366.219922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10367.230995] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10376.675304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.690455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10376.706718] Lustre: Skipped 2 previous similar messages [10376.715755] Lustre: Skipped 2 previous similar messages [10378.520942] Lustre: server umount lustre-MDT0000 complete [10381.649539] LustreError: 358432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393714 with bad export cookie 15614714827040312234 [10381.657559] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10381.662248] LustreError: 358432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10381.802990] LustreError: 358447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10381.823971] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.986052] Lustre: server umount lustre-MDT0001 complete [10387.172233] Lustre: server umount lustre-OST0000 complete [10397.583619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10397.898434] LustreError: 362159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.912246] LustreError: 362159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10397.955615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10401.697870] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10403.300585] LustreError: 362160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.396823] LustreError: 362159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.623546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.921875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10413.116787] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10419.455236] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10423.866792] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10429.824831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10430.119593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10435.785739] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10442.438799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10442.764291] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10455.009399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10455.019633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.028230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10458.081282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.085331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10458.093191] Lustre: Skipped 1 previous similar message [10458.116932] Lustre: Skipped 2 previous similar messages [10459.800614] Lustre: server umount lustre-MDT0000 complete [10462.517513] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393795 with bad export cookie 15614714827040313347 [10462.518209] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10462.529099] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10462.861892] Lustre: server umount lustre-MDT0001 complete [10475.522479] Lustre: server umount lustre-OST0000 complete [10489.322237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.705554] LustreError: 366069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.786490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10494.057155] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10501.603306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.837953] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10512.561171] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10517.795950] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10524.400995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.847550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10524.855230] Lustre: Skipped 1 previous similar message [10531.628774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10535.092715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10538.251522] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10545.636774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10545.640522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10545.652126] Lustre: Skipped 1 previous similar message [10545.664977] Lustre: Skipped 1 previous similar message [10549.007773] Lustre: server umount lustre-OST0000 complete [10553.315449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10553.328463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10555.875724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.892345] Lustre: Skipped 1 previous similar message [10566.120193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10566.130437] Lustre: Skipped 5 previous similar messages [10567.135271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10567.292421] Lustre: server umount lustre-MDT0000 complete [10571.075448] LustreError: 366049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393903 with bad export cookie 15614714827040314369 [10571.081686] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10571.086843] LustreError: 366049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10571.475499] Lustre: server umount lustre-MDT0001 complete [10582.534293] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [10585.490856] Key type lgssc unregistered [10585.814565] LNet: 369601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10585.821850] LNetError: 369601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10586.855730] LNet: Removed LNI 192.168.203.159@tcp [10587.830759] Key type .llcrypt unregistered [10587.832740] Key type ._llcrypt unregistered [10612.332397] Key type ._llcrypt registered [10612.339310] Key type .llcrypt registered [10612.488776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [10613.840334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10613.874981] alg: No test for adler32 (adler32-zlib) [10614.946693] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [10615.201034] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [10616.863510] Key type lgssc registered [10618.025293] Lustre: Echo OBD driver; http://www.lustre.org/ [10625.518282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10625.534703] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.542285] LustreError: 371055:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10625.556686] LustreError: 371055:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10625.562655] LustreError: 371055:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10625.569866] LustreError: 371055:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10625.575974] LustreError: 371055:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10633.080937] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.091960] LustreError: 371453:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10633.110211] LustreError: 371453:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10633.116705] LustreError: 371453:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10633.120639] LustreError: 371453:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10633.130733] LustreError: 371453:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10651.863740] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [10654.448294] Key type lgssc unregistered [10654.772158] LNet: 372456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10654.786132] LNetError: 372456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10654.801418] LNet: Removed LNI 192.168.203.159@tcp [10655.615417] Key type .llcrypt unregistered [10655.622440] Key type ._llcrypt unregistered [10673.863268] Key type ._llcrypt registered [10673.867246] Key type .llcrypt registered [10673.982813] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [10689.776506] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [10690.816381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10690.841541] alg: No test for adler32 (adler32-zlib) [10691.874295] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [10692.195762] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [10693.895226] Key type lgssc registered [10695.100253] Lustre: Echo OBD driver; http://www.lustre.org/ [10734.989813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.040640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.380914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10736.420841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10736.536455] Lustre: lustre-MDT0000: new disk, initializing [10736.647261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.676642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10741.021549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10751.568216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.710584] Lustre: 375618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.770634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10751.775646] Lustre: Skipped 1 previous similar message [10751.828218] Lustre: lustre-MDT0001: new disk, initializing [10751.963554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10752.031551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10752.057052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10756.579650] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10765.791950] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10771.375382] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10777.298865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.599366] Lustre: lustre-OST0000: new disk, initializing [10777.613968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10777.620533] Lustre: 377587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10777.716063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10784.258637] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10784.883455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10784.902900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10785.098209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10794.157130] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10799.131644] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10799.373440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10804.150815] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10804.370936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10810.854985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10810.855706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.869373] Lustre: Skipped 1 previous similar message [10810.871175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.880254] Lustre: Skipped 1 previous similar message [10812.504400] Lustre: server umount lustre-OST0000 complete [10815.969056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.975802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.990812] Lustre: Skipped 1 previous similar message [10816.005582] Lustre: Skipped 1 previous similar message [10821.090610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10821.100306] Lustre: Skipped 1 previous similar message [10826.208257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10826.214592] Lustre: Skipped 1 previous similar message [10830.303207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10830.306843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10830.321867] Lustre: Skipped 1 previous similar message [10830.603885] Lustre: server umount lustre-MDT0000 complete [10834.465260] LustreError: 375608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394167 with bad export cookie 6177384825263074764 [10834.466128] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10834.474898] LustreError: 375608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10834.846139] Lustre: server umount lustre-MDT0001 complete [10843.180643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10844.672480] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:22:55 (1787394175) [10851.314954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.829221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10856.789537] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10856.929223] LustreError: 380047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.939945] LustreError: 380047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10862.053198] LustreError: 380048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.555255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.173071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10872.128498] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10879.979974] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10885.583895] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10892.439536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.761394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10898.947931] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10906.356410] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10913.713336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10913.857620] Lustre: lustre-OST0001: new disk, initializing [10913.864707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10913.872581] Lustre: 382886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10913.960425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10920.631753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10920.653214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10920.801527] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [10920.818491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10930.501835] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10936.295386] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.295702] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10936.308626] Lustre: Skipped 1 previous similar message [10936.309520] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10936.340651] LustreError: Skipped 2 previous similar messages [10940.397172] Lustre: server umount lustre-OST0001 complete [10941.414977] LustreError: 381871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.530538] LustreError: 381870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.557865] LustreError: 381870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10951.650170] LustreError: 381871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.676600] LustreError: 381871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10958.308702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10958.321377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.339959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10958.348616] Lustre: Skipped 2 previous similar messages [10961.379102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10961.889218] LustreError: 382693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.913226] LustreError: 382693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10963.285804] Lustre: server umount lustre-OST0000 complete [10967.011935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.029717] Lustre: Skipped 2 previous similar messages [10981.343386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10981.573390] Lustre: server umount lustre-MDT0000 complete [10985.055703] LustreError: 380028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394317 with bad export cookie 6177384825263075646 [10985.057519] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10985.062799] LustreError: 380028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10991.441914] Lustre: server umount lustre-MDT0001 complete [11002.685450] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11005.855847] Key type lgssc unregistered [11006.147368] LNet: 384872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11006.158515] LNetError: 384872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11006.179293] LNet: Removed LNI 192.168.203.159@tcp [11007.043814] Key type .llcrypt unregistered [11007.051111] Key type ._llcrypt unregistered [11057.426749] Key type ._llcrypt registered [11057.432975] Key type .llcrypt registered [11057.578511] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [11058.285184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11058.432793] alg: No test for adler32 (adler32-zlib) [11059.532931] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11059.793832] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [11061.543922] Key type lgssc registered [11062.819686] Lustre: Echo OBD driver; http://www.lustre.org/ [11070.876792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11070.892323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.226947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11072.265761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11072.473396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11076.539317] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11085.094496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.191488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11085.219712] Lustre: 387676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11085.525601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11089.555629] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11097.246459] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11102.990326] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11106.274675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11106.280524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.292170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11108.832079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11108.834751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.234039] Lustre: server umount lustre-MDT0000 complete [11114.588526] LustreError: 387669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394447 with bad export cookie 4847546785808899263 [11114.593449] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11114.600362] LustreError: 387669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11114.993747] Lustre: server umount lustre-MDT0001 complete [11124.153729] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:27:34 (1787394454) [11130.432484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.849377] LustreError: 390187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.866323] LustreError: 390187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11130.909481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11135.271403] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11135.975204] LustreError: 390188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.092210] LustreError: 390187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11143.967798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.511474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11148.926579] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11156.151075] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11162.245836] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11169.165308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.356212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11169.584345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11175.518928] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11182.766994] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11189.878919] Lustre: Permanently deactivating lustre-OST0000 [11189.911337] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11197.475646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11198.783754] LustreError: 393174:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11198.807129] LustreError: 393174:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11198.812926] LustreError: 393174:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11202.005983] LustreError: 393223:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11202.012916] LustreError: 393223:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11209.821287] Lustre: server umount lustre-OST0000 complete [11212.726163] LustreError: 393473:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11212.737189] LustreError: 393473:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11215.328652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11215.343967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11215.353845] Lustre: Skipped 1 previous similar message [11215.366388] Lustre: Skipped 2 previous similar messages [11220.448740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11220.454215] Lustre: Skipped 1 previous similar message [11224.650899] Lustre: server umount lustre-MDT0000 complete [11225.572111] LustreError: 392320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.595166] LustreError: 392320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11228.734251] LustreError: 390168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394561 with bad export cookie 4847546785808899844 [11228.743404] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.748535] LustreError: 390168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11238.573382] Lustre: server umount lustre-MDT0001 complete [11246.753562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11250.765949] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11253.513316] LustreError: 394633:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11260.050318] Lustre: 394972:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.159@tcp [11262.073639] Lustre: server umount MGS complete [11273.192193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11273.627928] LustreError: 395696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.650745] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11273.657921] Lustre: Skipped 1 previous similar message [11273.720312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11278.109435] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11283.947208] LustreError: 395697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.958295] LustreError: 395697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11286.298136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11286.492611] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11290.817621] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11298.174439] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11302.734102] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11304.963311] Lustre: Permanently reactivating lustre-OST0000 [11304.976986] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11304.988602] Lustre: Skipped 2 previous similar messages [11331.552105] LustreError: 397442:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11340.407716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11340.970635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11340.983462] Lustre: Skipped 1 previous similar message [11347.186754] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11355.309441] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11362.798781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.803916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11362.821104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11362.826661] LustreError: Skipped 1 previous similar message [11362.849758] Lustre: Skipped 1 previous similar message [11363.346614] Lustre: server umount lustre-OST0000 complete [11367.908111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11367.934610] Lustre: Skipped 2 previous similar messages [11373.035792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11373.038462] Lustre: Skipped 2 previous similar messages [11373.843676] Lustre: server umount lustre-MDT0000 complete [11377.836546] LustreError: 395677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394710 with bad export cookie 4847546785808900894 [11377.844082] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11377.854517] LustreError: 395677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11378.357270] Lustre: server umount lustre-MDT0001 complete [11389.948208] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11392.900857] Key type lgssc unregistered [11393.198634] LNet: 400015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11393.204863] LNetError: 400015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11393.217158] LNet: Removed LNI 192.168.203.159@tcp [11393.915137] Key type .llcrypt unregistered [11393.919306] Key type ._llcrypt unregistered [11413.364640] Key type ._llcrypt registered [11413.366546] Key type .llcrypt registered [11413.480627] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [11430.096627] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [11431.592289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11431.629831] alg: No test for adler32 (adler32-zlib) [11432.885814] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11433.241106] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [11435.023908] Key type lgssc registered [11436.913320] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.495473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11473.515157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.805836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11474.834654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11474.879175] Lustre: lustre-MDT0000: new disk, initializing [11474.945522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11474.958706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11478.950246] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11488.633510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11488.733810] Lustre: 403173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11488.789178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11488.795215] Lustre: Skipped 1 previous similar message [11488.890650] Lustre: lustre-MDT0001: new disk, initializing [11488.993847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11489.047440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11489.063201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11493.427198] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11502.116452] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11507.870755] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11514.218369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11514.446000] Lustre: lustre-OST0000: new disk, initializing [11514.449737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11514.457547] Lustre: 405139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11514.551037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11520.386497] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11521.623922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11521.638674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11521.795861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11531.060156] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11536.630549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11536.962852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11541.765571] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11542.023372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11547.624402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11547.638885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11547.651800] Lustre: Skipped 1 previous similar message [11547.666806] Lustre: Skipped 1 previous similar message [11550.458322] Lustre: server umount lustre-OST0000 complete [11555.808379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11555.819973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.840754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11557.858464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11557.858497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.613569] Lustre: server umount lustre-MDT0000 complete [11563.952335] LustreError: 403165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394896 with bad export cookie 14935382195504984483 [11563.956762] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11563.966167] LustreError: 403165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11564.286878] Lustre: server umount lustre-MDT0001 complete [11572.132759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11574.042858] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:35:04 (1787394904) [11581.562581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11581.981610] LustreError: 407599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11581.992035] LustreError: 407599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11582.028396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11586.985215] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11587.049590] LustreError: 407600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.077319] LustreError: 407600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11591.153268] LustreError: 407599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.414798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.810426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11599.253502] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11606.233415] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11611.268786] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11617.467346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11617.812981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11623.577665] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11630.568714] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11632.347726] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11635.020954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11641.315340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11641.320985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.351585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.354354] Lustre: Skipped 1 previous similar message [11642.354723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11646.272740] Lustre: server umount lustre-OST0000 complete [11652.065656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11652.074941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.088946] Lustre: Skipped 1 previous similar message [11652.091706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11652.101320] Lustre: Skipped 2 previous similar messages [11662.817100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11662.824844] Lustre: Skipped 4 previous similar messages [11664.351122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11664.617856] Lustre: server umount lustre-MDT0000 complete [11667.938156] LustreError: 407595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.955654] LustreError: 407595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11668.391713] LustreError: 407579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395001 with bad export cookie 14935382195504985365 [11668.404497] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.407177] LustreError: 407579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11668.731171] Lustre: server umount lustre-MDT0001 complete [11680.341838] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11683.275814] Key type lgssc unregistered [11683.616282] LNet: 411324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11683.627419] LNetError: 411324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11683.661976] LNet: Removed LNI 192.168.203.159@tcp [11684.591152] Key type .llcrypt unregistered [11684.596922] Key type ._llcrypt unregistered [11706.010730] Key type ._llcrypt registered [11706.014921] Key type .llcrypt registered [11706.138552] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11706.445622] Key type .llcrypt unregistered [11706.449517] Key type ._llcrypt unregistered [11720.721144] Key type ._llcrypt registered [11720.725720] Key type .llcrypt registered [11720.846494] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [11721.970760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.989138] alg: No test for adler32 (adler32-zlib) [11723.106393] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11723.406436] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [11725.119218] Key type lgssc registered [11726.260199] Lustre: Echo OBD driver; http://www.lustre.org/ [11733.520665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11733.543299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.089943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11739.322337] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11744.230769] LustreError: 413516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.243469] LustreError: 413516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11747.000614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11747.422300] LustreError: 413517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.459317] LustreError: 413517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11747.477757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11752.928597] LustreError: 413516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.962136] LustreError: 413516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11754.499655] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11758.049097] LustreError: 413517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.080699] LustreError: 413517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11760.122420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11762.504307] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11765.219621] LustreError: 413513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.255523] LustreError: 413513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11769.551466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.774619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11773.750765] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11783.531357] Lustre: server umount lustre-OST0000 complete [11788.779812] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11788.805511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11788.826779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11790.306721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11790.307330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.428145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11795.459486] Lustre: Skipped 2 previous similar messages [11797.647909] Lustre: server umount lustre-MDT0000 complete [11800.545866] LustreError: 414760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.572522] LustreError: 414760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11801.569974] Lustre: 412829:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395118/real 1787395118] req@00000000c40f3c90 x1874219558267264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787395134 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11801.570924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.596160] Lustre: 412829:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11802.644754] LustreError: 413496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395135 with bad export cookie 8662460580171150441 [11802.660487] LustreError: 413496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11802.664770] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11802.992238] Lustre: server umount lustre-MDT0001 complete [11814.474957] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11817.273715] Key type lgssc unregistered [11817.603019] LNet: 416666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11817.607862] LNetError: 416666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11817.623626] LNet: Removed LNI 192.168.203.159@tcp [11818.309516] Key type .llcrypt unregistered [11818.311141] Key type ._llcrypt unregistered [11844.803492] Key type ._llcrypt registered [11844.805454] Key type .llcrypt registered [11844.874480] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [11846.081315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.104371] alg: No test for adler32 (adler32-zlib) [11847.142411] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11847.330847] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [11849.007176] Key type lgssc registered [11850.259171] Lustre: Echo OBD driver; http://www.lustre.org/ [11858.438671] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11858.460715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11874.079583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11879.201052] LustreError: 418056:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [11905.823426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11911.272897] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11917.830761] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11923.663161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.080031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11924.158903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11927.729980] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11929.570070] LustreError: 418082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.589350] LustreError: 418082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11934.694749] LustreError: 418081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11934.880600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.177164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11939.419408] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [11946.667960] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11951.864661] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11959.788498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11959.797397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.818447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11960.803869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11960.811975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11960.825388] LustreError: 418072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11960.957660] Lustre: server umount lustre-OST0000 complete [11965.921769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11965.932774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.966345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11967.381971] Lustre: server umount lustre-MDT0000 complete [11971.007213] LustreError: 418063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395303 with bad export cookie 12691338494297531276 [11971.011386] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.021663] LustreError: 418063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11971.477799] Lustre: server umount lustre-MDT0001 complete [11981.801520] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [11984.941928] Key type lgssc unregistered [11985.269535] LNet: 421622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.281686] LNetError: 421622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11985.312373] LNet: Removed LNI 192.168.203.159@tcp [11986.243936] Key type .llcrypt unregistered [11986.250472] Key type ._llcrypt unregistered [12010.827779] Key type ._llcrypt registered [12010.830624] Key type .llcrypt registered [12010.924992] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [12011.977484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12012.028185] alg: No test for adler32 (adler32-zlib) [12013.125400] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12013.336453] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [12014.999477] Key type lgssc registered [12016.016409] Lustre: Echo OBD driver; http://www.lustre.org/ [12023.233565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12023.255732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12024.683147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12029.166932] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12035.049987] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12035.074554] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12037.361285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12037.750667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.949213] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12048.771698] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12054.064166] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12062.039527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12062.578540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.766909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12070.876591] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12080.359334] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12085.220244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.247566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12085.249653] Lustre: Skipped 2 previous similar messages [12085.267327] Lustre: Skipped 2 previous similar messages [12087.176614] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [12090.354180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12090.362857] Lustre: Skipped 2 previous similar messages [12090.603421] Lustre: server umount lustre-MDT0000 complete [12092.271139] LustreError: 423034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.154356] LustreError: 423021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.59@tcp arrived at 1787395426 with bad export cookie 17116854311465109192 [12093.181931] LustreError: 423021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12095.185646] LustreError: 423020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395428 with bad export cookie 17116854311465108296 [12095.199438] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12095.210483] LustreError: 423020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12095.457039] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.470973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12095.507181] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12095.532397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.586595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.587423] LustreError: 423036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.615765] LustreError: 423036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12101.598824] Lustre: server umount lustre-OST0000 complete [12105.697254] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.733915] LustreError: 423040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12108.838246] Lustre: server umount lustre-MDT0001 complete [12119.642541] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [12122.469687] Key type lgssc unregistered [12122.745770] LNet: 426726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12122.762641] LNetError: 426726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12123.824716] LNet: Removed LNI 192.168.203.159@tcp [12124.870136] Key type .llcrypt unregistered [12124.876778] Key type ._llcrypt unregistered [12150.899388] Key type ._llcrypt registered [12150.902607] Key type .llcrypt registered [12151.021720] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [12151.903795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12151.976366] alg: No test for adler32 (adler32-zlib) [12152.998425] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12153.316583] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [12155.017094] Key type lgssc registered [12156.403581] Lustre: Echo OBD driver; http://www.lustre.org/ [12164.234079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12164.249674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.725228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12170.159424] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12175.841433] LustreError: 428143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.877812] LustreError: 428143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12178.688422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.982278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12183.555268] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12190.565241] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12195.917182] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12202.395685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12202.807826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12208.867376] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12210.993458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12215.742591] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12221.409668] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.419362] Lustre: Skipped 1 previous similar message [12221.420441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12221.435375] Lustre: Skipped 1 previous similar message [12223.358273] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [12226.528773] Lustre: server umount lustre-MDT0001 complete [12226.533414] LustreError: 429146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.649476] LustreError: 428143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.676258] LustreError: 428143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12236.771236] LustreError: 428144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.772663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12236.775684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12236.777419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12238.734953] Lustre: server umount lustre-OST0000 complete [12242.062305] Lustre: server umount lustre-MDT0000 complete [12252.790919] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [12255.693873] Key type lgssc unregistered [12256.053578] LNet: 431825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12256.067956] LNetError: 431825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12256.088395] LNet: Removed LNI 192.168.203.159@tcp [12257.066942] Key type .llcrypt unregistered [12257.069582] Key type ._llcrypt unregistered [12279.236224] Key type ._llcrypt registered [12279.240403] Key type .llcrypt registered [12279.363780] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [12292.569822] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [12293.498171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12293.613899] alg: No test for adler32 (adler32-zlib) [12294.732780] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12295.023602] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [12296.751164] Key type lgssc registered [12297.727715] Lustre: Echo OBD driver; http://www.lustre.org/ [12328.365301] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [12341.483041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12341.512573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.739867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12342.777815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12342.853824] Lustre: lustre-MDT0000: new disk, initializing [12342.906753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12342.924340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12346.699420] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12361.648675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12362.003403] Lustre: lustre-OST0000: new disk, initializing [12362.008288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12362.011186] Lustre: Skipped 1 previous similar message [12362.018276] Lustre: 436838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12362.122450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12368.300040] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12368.514418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12368.525595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12368.569058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12378.349400] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12393.873927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.080379] Lustre: 435820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12394.191520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12394.299909] Lustre: lustre-MDT0001: new disk, initializing [12394.486820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12394.547162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12394.571890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12399.158528] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12408.057860] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12408.232959] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12412.695953] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12412.897940] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12418.448235] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [12421.096490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12421.103474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12421.115645] Lustre: Skipped 1 previous similar message [12421.116041] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12421.149293] Lustre: Skipped 1 previous similar message [12423.581249] Lustre: server umount lustre-MDT0001 complete [12428.641103] LustreError: 438164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.672204] LustreError: 438164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12431.329574] LustreError: 435826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12431.357221] LustreError: 435826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12436.454157] LustreError: 438164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.466702] LustreError: 438164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12441.573184] LustreError: 435828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12441.590962] LustreError: 435828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12445.664530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12445.675665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12445.698816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12446.688995] LustreError: 435831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.085784] Lustre: server umount lustre-OST0000 complete [12466.143612] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12466.378168] Lustre: server umount lustre-MDT0000 complete [12482.332970] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [12485.593369] Key type lgssc unregistered [12485.953242] LNet: 440585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12485.963720] LNetError: 440585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12485.991837] LNet: Removed LNI 192.168.203.159@tcp [12487.033365] Key type .llcrypt unregistered [12487.037234] Key type ._llcrypt unregistered [12528.094879] Key type ._llcrypt registered [12528.098118] Key type .llcrypt registered [12528.248237] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [12528.518612] Key type .llcrypt unregistered [12528.525306] Key type ._llcrypt unregistered [12546.249050] Key type ._llcrypt registered [12546.250588] Key type .llcrypt registered [12546.379986] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [12547.667372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12547.774144] alg: No test for adler32 (adler32-zlib) [12548.843425] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12549.144446] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [12550.826210] Key type lgssc registered [12552.083698] Lustre: Echo OBD driver; http://www.lustre.org/ [12576.194791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12576.227829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12577.482564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12577.506220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12577.578154] Lustre: lustre-MDT0000: new disk, initializing [12577.644928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12577.672778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12581.864545] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12591.501571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.682728] Lustre: 443962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12591.753299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12591.757272] Lustre: Skipped 1 previous similar message [12591.809455] Lustre: lustre-MDT0001: new disk, initializing [12591.898262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12591.932871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12591.942652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12596.145784] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12605.112446] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12610.692636] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12616.882949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12617.128421] Lustre: lustre-OST0000: new disk, initializing [12617.132048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12617.143407] Lustre: 445931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12617.232684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.814932] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12622.921025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12622.929245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12623.161681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12632.039675] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12638.691179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.695182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12638.708713] Lustre: Skipped 1 previous similar message [12638.721306] Lustre: Skipped 1 previous similar message [12643.296270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12643.314349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.323874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12644.593911] Lustre: server umount lustre-MDT0000 complete [12648.227482] LustreError: 443956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395981 with bad export cookie 7353708711878080955 [12648.237173] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12648.237483] LustreError: 443956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12648.669040] Lustre: server umount lustre-MDT0001 complete [12662.902805] Lustre: server umount lustre-OST0000 complete [12672.934249] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:53:23 (1787396003) [12680.836544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12681.459863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12686.130946] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12686.827289] LustreError: 447960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.840900] LustreError: 447960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12691.938743] LustreError: 447959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.536696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.972359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12699.402466] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12706.094860] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12711.043837] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12730.154396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12730.418866] Lustre: Found index 0 for lustre-OST0000, updating log [12730.662074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12736.808346] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [12743.919315] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12752.173458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12761.567977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12761.576491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12761.597020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12761.606046] Lustre: Skipped 2 previous similar messages [12768.743813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12768.751637] Lustre: Skipped 4 previous similar messages [12772.319562] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12772.582306] Lustre: server umount lustre-OST0000 complete [12787.893126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12787.915696] Lustre: Skipped 1 previous similar message [12787.924573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12792.288226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12792.291373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12794.321399] Lustre: server umount lustre-MDT0000 complete [12797.613457] LustreError: 447939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396130 with bad export cookie 7353708711878082348 [12797.617594] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12797.628816] LustreError: 447939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12797.965863] Lustre: server umount lustre-MDT0001 complete [12800.031611] LNet: 1 local NIs in recovery (showing 1): 192.168.203.159@tcp [12805.456048] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:55:36 (1787396136) [13418.609132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13419.011859] LustreError: 454403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.113062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13422.662103] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13424.099611] LustreError: 454404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.194158] LustreError: 454403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.676107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13428.950983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13432.553701] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13437.637830] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13441.429869] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13459.046437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13459.249773] Lustre: Found index 0 for lustre-OST0000, updating log [13459.573462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13464.497396] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13466.744403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13470.237582] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13472.226897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13472.232960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13472.252242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13472.256987] Lustre: Skipped 3 previous similar messages [13477.983351] LNet: 1 local NIs in recovery (showing 1): 192.168.203.159@tcp [13478.168172] Lustre: server umount lustre-OST0000 complete [13481.046291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13481.065948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13481.074424] Lustre: Skipped 1 previous similar message [13483.108525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13483.116605] Lustre: Skipped 3 previous similar messages [13487.203139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13487.214029] Lustre: Skipped 7 previous similar messages [13487.374511] Lustre: server umount lustre-MDT0000 complete [13488.233615] LustreError: 454398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13488.253771] LustreError: 454398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13490.397401] LustreError: 455136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396823 with bad export cookie 7353708711878083433 [13490.405436] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13490.411991] LustreError: 455136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13490.768153] Lustre: server umount lustre-MDT0001 complete [13517.119438] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [13524.893372] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [13554.346350] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:08:05 (1787396885) [13570.832940] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [13573.301542] Key type lgssc unregistered [13573.589733] LNet: 461021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13573.600248] LNetError: 461021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13682.155621] LNet: Removed LNI 192.168.203.159@tcp [13682.989501] Key type .llcrypt unregistered [13682.994124] Key type ._llcrypt unregistered [13699.436599] Key type ._llcrypt registered [13699.438417] Key type .llcrypt registered [13699.494947] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [13700.287059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13700.455933] alg: No test for adler32 (adler32-zlib) [13701.591661] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [13701.723488] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [13703.383221] Key type lgssc registered [13704.325542] Lustre: Echo OBD driver; http://www.lustre.org/ [13708.093217] LNet: Added LNI 192.168.203.159@tcp100 [8/256/0/180] [13708.800037] LNet: Added LNI 192.168.203.159@tcp101 [8/256/0/180] [13709.498855] LNet: Added LNI 192.168.203.159@tcp102 [8/256/0/180] [13710.170738] LNet: Added LNI 192.168.203.159@tcp103 [8/256/0/180] [13710.798877] LNet: Added LNI 192.168.203.159@tcp104 [8/256/0/180] [13711.602140] LNet: Added LNI 192.168.203.159@tcp105 [8/256/0/180] [13712.231599] LNet: Added LNI 192.168.203.159@tcp106 [8/256/0/180] [13713.207697] LNet: Added LNI 192.168.203.159@tcp107 [8/256/0/180] [13713.794574] LNet: Added LNI 192.168.203.159@tcp108 [8/256/0/180] [13714.540623] LNet: Added LNI 192.168.203.159@tcp109 [8/256/0/180] [13715.254181] LNet: Added LNI 192.168.203.159@tcp110 [8/256/0/180] [13715.914341] LNet: Added LNI 192.168.203.159@tcp111 [8/256/0/180] [13716.585326] LNet: Added LNI 192.168.203.159@tcp112 [8/256/0/180] [13717.238579] LNet: Added LNI 192.168.203.159@tcp113 [8/256/0/180] [13717.921144] LNet: Added LNI 192.168.203.159@tcp114 [8/256/0/180] [13718.646500] LNet: Added LNI 192.168.203.159@tcp115 [8/256/0/180] [13719.343568] LNet: Added LNI 192.168.203.159@tcp116 [8/256/0/180] [13720.109257] LNet: Added LNI 192.168.203.159@tcp117 [8/256/0/180] [13720.912571] LNet: Added LNI 192.168.203.159@tcp118 [8/256/0/180] [13721.617846] LNet: Added LNI 192.168.203.159@tcp119 [8/256/0/180] [13722.482363] LNet: Added LNI 192.168.203.159@tcp120 [8/256/0/180] [13723.169827] LNet: Added LNI 192.168.203.159@tcp121 [8/256/0/180] [13723.879519] LNet: Added LNI 192.168.203.159@tcp122 [8/256/0/180] [13724.647743] LNet: Added LNI 192.168.203.159@tcp123 [8/256/0/180] [13725.345298] LNet: Added LNI 192.168.203.159@tcp124 [8/256/0/180] [13726.039729] LNet: Added LNI 192.168.203.159@tcp125 [8/256/0/180] [13726.796891] LNet: Added LNI 192.168.203.159@tcp126 [8/256/0/180] [13727.521154] LNet: Added LNI 192.168.203.159@tcp127 [8/256/0/180] [13728.293722] LNet: Added LNI 192.168.203.159@tcp128 [8/256/0/180] [13729.130802] LNet: Added LNI 192.168.203.159@tcp129 [8/256/0/180] [13729.921507] LNet: Added LNI 192.168.203.159@tcp130 [8/256/0/180] [13730.745611] LNet: Added LNI 192.168.203.159@tcp131 [8/256/0/180] [13731.540270] LNet: Added LNI 192.168.203.159@tcp132 [8/256/0/180] [13732.297952] LNet: Added LNI 192.168.203.159@tcp133 [8/256/0/180] [13733.038612] LNet: Added LNI 192.168.203.159@tcp134 [8/256/0/180] [13733.777405] LNet: Added LNI 192.168.203.159@tcp135 [8/256/0/180] [13739.320556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13739.343093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13740.436767] Lustre: 464051:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [13740.512598] LNetError: 464082:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13740.544759] LNetError: Protocol error connecting to 192.168.203.159@tcp117 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13740.549506] LNetError: Skipped 2 previous similar messages [13740.597420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13740.623319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13740.696610] Lustre: lustre-MDT0000: new disk, initializing [13740.783935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13740.800891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13741.605641] LNetError: 464084:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13741.633107] LNetError: 464084:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13741.652847] LNetError: Protocol error connecting to 192.168.203.159@tcp108 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13743.978121] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13753.276290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13753.439335] Lustre: 464069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13753.507751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13753.514252] Lustre: Skipped 1 previous similar message [13753.630698] Lustre: lustre-MDT0001: new disk, initializing [13753.683303] Lustre: 464871:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [13753.688289] Lustre: 464871:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13753.795458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13753.832686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13753.842034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13758.162869] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13759.017262] Lustre: 465176:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [13759.028285] Lustre: 465176:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13766.426873] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13771.130637] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13771.365114] LNetError: 464074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13771.380447] LNetError: Protocol error connecting to 192.168.203.159@tcp108 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13773.426561] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13773.438945] LNetError: Protocol error connecting to 192.168.203.159@tcp129 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13777.509671] LNetError: 461512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13777.520501] LNetError: 461512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13777.532707] LNetError: Protocol error connecting to 192.168.203.159@tcp117 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13777.551294] LNetError: Skipped 1 previous similar message [13777.743133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13778.084351] Lustre: lustre-OST0000: new disk, initializing [13778.088788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13778.096801] Lustre: 466043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13778.193915] Lustre: 466048:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [13778.198872] Lustre: 466048:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13778.243348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13783.590945] Lustre: 466380:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [13783.605741] Lustre: 466380:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13783.651928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13783.665125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13783.734493] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [13783.793491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13785.699321] LNetError: 464080:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13785.713199] LNetError: 464080:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13785.720460] LNetError: Protocol error connecting to 192.168.203.159@tcp129 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13785.733167] LNetError: Skipped 2 previous similar messages [13791.540985] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13799.392149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13799.394436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13799.406063] LustreError: Skipped 1 previous similar message [13799.407045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13799.450227] Lustre: Skipped 1 previous similar message [13802.080662] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [13802.092645] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13802.112982] LNetError: Protocol error connecting to 192.168.203.159@tcp117 at host 192.168.203.159:988: is it running a compatible version of Lustre? [13802.137933] LNetError: Skipped 6 previous similar messages [13804.528519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13804.537635] Lustre: Skipped 2 previous similar messages [13804.734560] Lustre: server umount lustre-OST0000 complete [13809.633781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13809.637344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13809.665981] Lustre: Skipped 1 previous similar message [13809.676425] Lustre: Skipped 1 previous similar message [13814.755781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13814.769883] Lustre: Skipped 1 previous similar message [13815.295393] Lustre: server umount lustre-MDT0000 complete [13820.327084] LustreError: 464061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397153 with bad export cookie 5381749125811314236 [13820.342212] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13820.343032] LustreError: 464061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13821.082890] Lustre: server umount lustre-MDT0001 complete [13834.054941] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [13837.399384] Key type lgssc unregistered [13837.871498] LNet: 467955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13837.887692] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13837.899716] LNet: Removed LNI 192.168.203.159@tcp [13838.943622] LNet: Removed LNI 192.168.203.159@tcp100 [13838.947503] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13838.963590] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13838.973097] LNet: Removed LNI 192.168.203.159@tcp101 [13840.031163] LNet: Removed LNI 192.168.203.159@tcp102 [13840.040836] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13840.049866] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13840.069673] LNet: Removed LNI 192.168.203.159@tcp103 [13841.119261] LNet: Removed LNI 192.168.203.159@tcp104 [13842.143165] LNet: Removed LNI 192.168.203.159@tcp105 [13842.152327] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13842.167892] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13842.184250] LNet: Removed LNI 192.168.203.159@tcp106 [13843.231208] LNet: Removed LNI 192.168.203.159@tcp107 [13844.255277] LNet: Removed LNI 192.168.203.159@tcp108 [13845.279655] LNet: Removed LNI 192.168.203.159@tcp109 [13846.303328] LNet: Removed LNI 192.168.203.159@tcp110 [13846.312386] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13846.323305] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13846.348432] LNet: Removed LNI 192.168.203.159@tcp111 [13847.391943] LNet: Removed LNI 192.168.203.159@tcp112 [13848.415254] LNet: Removed LNI 192.168.203.159@tcp113 [13849.504166] LNet: Removed LNI 192.168.203.159@tcp114 [13850.591167] LNet: Removed LNI 192.168.203.159@tcp115 [13851.615393] LNet: Removed LNI 192.168.203.159@tcp116 [13852.639226] LNet: Removed LNI 192.168.203.159@tcp117 [13853.666090] LNet: Removed LNI 192.168.203.159@tcp118 [13854.687131] LNet: Removed LNI 192.168.203.159@tcp119 [13854.694378] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13854.702221] LNetError: 467955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13854.716090] LNet: Removed LNI 192.168.203.159@tcp120 [13855.775141] LNet: Removed LNI 192.168.203.159@tcp121 [13856.863150] LNet: Removed LNI 192.168.203.159@tcp122 [13857.887118] LNet: Removed LNI 192.168.203.159@tcp123 [13858.914451] LNet: Removed LNI 192.168.203.159@tcp124 [13859.935766] LNet: Removed LNI 192.168.203.159@tcp125 [13860.959248] LNet: Removed LNI 192.168.203.159@tcp126 [13861.983790] LNet: Removed LNI 192.168.203.159@tcp127 [13863.007352] LNet: Removed LNI 192.168.203.159@tcp128 [13864.031145] LNet: Removed LNI 192.168.203.159@tcp129 [13864.034697] LNet: Removed LNI 192.168.203.159@tcp130 [13864.041356] LNet: Removed LNI 192.168.203.159@tcp131 [13864.044164] LNet: Removed LNI 192.168.203.159@tcp132 [13864.054580] LNet: Removed LNI 192.168.203.159@tcp133 [13864.058326] LNet: Removed LNI 192.168.203.159@tcp134 [13864.076721] LNet: Removed LNI 192.168.203.159@tcp135 [13864.941323] Key type .llcrypt unregistered [13864.944231] Key type ._llcrypt unregistered [13891.076818] Key type ._llcrypt registered [13891.080395] Key type .llcrypt registered [13891.246767] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [13906.188409] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [13907.318808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13907.331978] alg: No test for adler32 (adler32-zlib) [13908.549612] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [13908.861364] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [13910.607714] Key type lgssc registered [13911.813767] Lustre: Echo OBD driver; http://www.lustre.org/ [13947.808561] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:14:38 (1787397278) [13965.076421] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [13967.711307] Key type lgssc unregistered [13967.956530] LNet: 472053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13967.968924] LNetError: 472053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13967.989689] LNet: Removed LNI 192.168.203.159@tcp [13968.777693] Key type .llcrypt unregistered [13968.782703] Key type ._llcrypt unregistered [13984.519237] Key type ._llcrypt registered [13984.521357] Key type .llcrypt registered [13984.619439] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [13985.953056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13985.987071] alg: No test for adler32 (adler32-zlib) [13987.100415] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [13987.446824] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [13989.175479] Key type lgssc registered [13990.351616] Lustre: Echo OBD driver; http://www.lustre.org/ [13998.779124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13998.794045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13999.981567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14000.042661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14000.105309] Lustre: lustre-MDT0000: new disk, initializing [14000.166208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14000.189468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14004.603842] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14016.116085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14016.403802] Lustre: 473264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14016.501128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14016.517179] Lustre: Skipped 1 previous similar message [14016.664741] Lustre: lustre-MDT0001: new disk, initializing [14016.862710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14016.921429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14016.944575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14022.212820] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14031.998360] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14037.511746] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14044.166933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14044.469577] Lustre: lustre-OST0000: new disk, initializing [14044.473705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14044.481600] Lustre: 475232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14044.596933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14050.907772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14050.920241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14051.053351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14051.340809] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14061.041391] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14065.827738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing lnet_if_list [14071.538233] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14071.573453] LNet: Added LNI 192.168.203.159@tcp10 [8/256/0/180] [14078.008957] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14078.033657] LNet: Added LNI 192.168.203.159@tcp11 [8/256/0/180] [14082.869693] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14082.905215] LNet: Added LNI 192.168.203.159@tcp12 [8/256/0/180] [14087.232851] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14087.269771] LNet: Added LNI 192.168.203.159@tcp13 [8/256/0/180] [14091.186439] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14091.234849] LNet: Added LNI 192.168.203.159@tcp14 [8/256/0/180] [14096.070671] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14096.097393] LNet: Added LNI 192.168.203.159@tcp15 [8/256/0/180] [14100.486628] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14100.513115] LNet: Added LNI 192.168.203.159@tcp16 [8/256/0/180] [14106.325151] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14106.349778] LNet: Added LNI 192.168.203.159@tcp17 [8/256/0/180] [14112.763419] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14112.791401] LNet: Added LNI 192.168.203.159@tcp18 [8/256/0/180] [14119.134584] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14119.197811] LNet: Added LNI 192.168.203.159@tcp19 [8/256/0/180] [14125.723416] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14125.786452] LNet: Added LNI 192.168.203.159@tcp20 [8/256/0/180] [14130.276826] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14130.333803] LNet: Added LNI 192.168.203.159@tcp21 [8/256/0/180] [14135.198427] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14135.232427] LNet: Added LNI 192.168.203.159@tcp22 [8/256/0/180] [14140.535658] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14140.575494] LNet: Added LNI 192.168.203.159@tcp23 [8/256/0/180] [14145.917630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14145.943871] LNet: Added LNI 192.168.203.159@tcp24 [8/256/0/180] [14151.751801] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14151.812847] LNet: Added LNI 192.168.203.159@tcp25 [8/256/0/180] [14158.612253] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14158.642181] LNet: Added LNI 192.168.203.159@tcp26 [8/256/0/180] [14165.074423] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14165.095545] LNet: Added LNI 192.168.203.159@tcp27 [8/256/0/180] [14171.361164] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14171.395937] LNet: Added LNI 192.168.203.159@tcp28 [8/256/0/180] [14177.802822] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14177.844039] LNet: Added LNI 192.168.203.159@tcp29 [8/256/0/180] [14184.450954] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14184.485321] LNet: Added LNI 192.168.203.159@tcp30 [8/256/0/180] [14191.393743] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14191.413666] LNet: Added LNI 192.168.203.159@tcp31 [8/256/0/180] [14197.477649] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14197.496771] LNet: Added LNI 192.168.203.159@tcp32 [8/256/0/180] [14204.202710] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14204.238647] LNet: Added LNI 192.168.203.159@tcp33 [8/256/0/180] [14211.704558] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14211.780532] LNet: Added LNI 192.168.203.159@tcp34 [8/256/0/180] [14219.184594] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14219.217646] LNet: Added LNI 192.168.203.159@tcp35 [8/256/0/180] [14225.542253] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14225.579350] LNet: Added LNI 192.168.203.159@tcp36 [8/256/0/180] [14233.044990] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14233.081693] LNet: Added LNI 192.168.203.159@tcp37 [8/256/0/180] [14240.501306] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14240.527442] LNet: Added LNI 192.168.203.159@tcp38 [8/256/0/180] [14249.586789] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14249.624434] LNet: Added LNI 192.168.203.159@tcp39 [8/256/0/180] [14260.971795] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14260.997080] LNet: Added LNI 192.168.203.159@tcp40 [8/256/0/180] [14268.856677] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14268.892229] LNet: Added LNI 192.168.203.159@tcp41 [8/256/0/180] [14277.007572] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14277.041764] LNet: Added LNI 192.168.203.159@tcp42 [8/256/0/180] [14284.541106] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14284.565592] LNet: Added LNI 192.168.203.159@tcp43 [8/256/0/180] [14294.191414] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14294.227184] LNet: Added LNI 192.168.203.159@tcp44 [8/256/0/180] [14304.370398] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14304.418796] LNet: Added LNI 192.168.203.159@tcp45 [8/256/0/180] [14312.488113] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14312.512765] LNet: Added LNI 192.168.203.159@tcp46 [8/256/0/180] [14321.528397] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14321.566723] LNet: Added LNI 192.168.203.159@tcp47 [8/256/0/180] [14329.666937] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14329.700337] LNet: Added LNI 192.168.203.159@tcp48 [8/256/0/180] [14336.759705] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14336.783960] LNet: Added LNI 192.168.203.159@tcp49 [8/256/0/180] [14344.384788] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14344.419584] LNet: Added LNI 192.168.203.159@tcp50 [8/256/0/180] [14363.618364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14363.630924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14363.634467] Lustre: Skipped 2 previous similar messages [14363.658904] Lustre: Skipped 2 previous similar messages [14368.743953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14368.749623] Lustre: Skipped 2 previous similar messages [14373.857695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14373.865090] Lustre: Skipped 2 previous similar messages [14375.402335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14375.611894] Lustre: server umount lustre-MDT0000 complete [14378.799265] LustreError: 473257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397711 with bad export cookie 12900281950414718565 [14378.800438] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14378.808926] LustreError: 473257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14378.981798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14379.155215] Lustre: server umount lustre-MDT0001 complete [14396.271498] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [14403.330751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14403.555192] LustreError: 472535:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9b12320 x1874221932391040/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14403.747594] LustreError: 475225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14403.755697] LustreError: 475225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14403.786400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14407.061748] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14408.938783] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14408.953176] LustreError: 475226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14414.307195] LustreError: 475225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14414.521615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14415.052698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14419.009849] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14419.125329] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14424.575382] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14429.778660] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14450.145313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14450.149313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14450.160253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14450.170409] Lustre: Skipped 1 previous similar message [14452.584276] Lustre: server umount lustre-OST0000 complete [14455.778219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14455.790261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14455.806937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14455.809560] Lustre: Skipped 1 previous similar message [14460.394911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14465.504168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14465.520763] Lustre: Skipped 3 previous similar messages [14470.111122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14470.297770] Lustre: server umount lustre-MDT0000 complete [14470.624716] LustreError: 499478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14470.639474] LustreError: 499478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14473.724806] LustreError: 473257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397806 with bad export cookie 12900281950414721148 [14473.732683] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14473.737577] LustreError: 473257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14474.099807] Lustre: server umount lustre-MDT0001 complete [14482.469471] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [14485.003254] Key type lgssc unregistered [14485.434941] LNet: 502253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14485.444143] LNetError: 502253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14485.452547] LNet: Removed LNI 192.168.203.159@tcp [14485.458690] LNet: Removed LNI 192.168.203.159@tcp10 [14485.463637] LNet: Removed LNI 192.168.203.159@tcp11 [14485.471554] LNet: Removed LNI 192.168.203.159@tcp12 [14485.482918] LNet: Removed LNI 192.168.203.159@tcp13 [14485.489130] LNet: Removed LNI 192.168.203.159@tcp14 [14485.493880] LNet: Removed LNI 192.168.203.159@tcp15 [14485.496262] LNet: Removed LNI 192.168.203.159@tcp16 [14485.499352] LNet: Removed LNI 192.168.203.159@tcp17 [14485.502210] LNet: Removed LNI 192.168.203.159@tcp18 [14485.505261] LNet: Removed LNI 192.168.203.159@tcp19 [14485.508502] LNet: Removed LNI 192.168.203.159@tcp20 [14485.511199] LNet: Removed LNI 192.168.203.159@tcp21 [14485.514698] LNet: Removed LNI 192.168.203.159@tcp22 [14485.522588] LNet: Removed LNI 192.168.203.159@tcp23 [14485.524612] LNet: Removed LNI 192.168.203.159@tcp24 [14485.529725] LNet: Removed LNI 192.168.203.159@tcp25 [14485.532780] LNet: Removed LNI 192.168.203.159@tcp26 [14485.535740] LNet: Removed LNI 192.168.203.159@tcp27 [14485.539159] LNet: Removed LNI 192.168.203.159@tcp28 [14485.543209] LNet: Removed LNI 192.168.203.159@tcp29 [14485.546243] LNet: Removed LNI 192.168.203.159@tcp30 [14485.548896] LNet: Removed LNI 192.168.203.159@tcp31 [14485.551583] LNet: Removed LNI 192.168.203.159@tcp32 [14485.554548] LNet: Removed LNI 192.168.203.159@tcp33 [14485.557463] LNet: Removed LNI 192.168.203.159@tcp34 [14485.560415] LNet: Removed LNI 192.168.203.159@tcp35 [14485.563321] LNet: Removed LNI 192.168.203.159@tcp36 [14485.566277] LNet: Removed LNI 192.168.203.159@tcp37 [14485.569251] LNet: Removed LNI 192.168.203.159@tcp38 [14485.572332] LNet: Removed LNI 192.168.203.159@tcp39 [14485.576399] LNet: Removed LNI 192.168.203.159@tcp40 [14485.581671] LNet: Removed LNI 192.168.203.159@tcp41 [14485.584592] LNet: Removed LNI 192.168.203.159@tcp42 [14485.587413] LNet: Removed LNI 192.168.203.159@tcp43 [14485.590535] LNet: Removed LNI 192.168.203.159@tcp44 [14485.593633] LNet: Removed LNI 192.168.203.159@tcp45 [14485.597932] LNet: Removed LNI 192.168.203.159@tcp46 [14485.601967] LNet: Removed LNI 192.168.203.159@tcp47 [14485.605113] LNet: Removed LNI 192.168.203.159@tcp48 [14485.607091] LNet: Removed LNI 192.168.203.159@tcp49 [14485.621279] LNet: Removed LNI 192.168.203.159@tcp50 [14486.379982] Key type .llcrypt unregistered [14486.383339] Key type ._llcrypt unregistered [14504.519825] Key type ._llcrypt registered [14504.522401] Key type .llcrypt registered [14504.625030] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [14518.012209] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [14519.187417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14519.217318] alg: No test for adler32 (adler32-zlib) [14520.268521] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [14520.512713] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [14522.199394] Key type lgssc registered [14523.624411] Lustre: Echo OBD driver; http://www.lustre.org/ [14553.343055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14553.356155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14554.583468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14554.622450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14554.696819] Lustre: lustre-MDT0000: new disk, initializing [14554.746323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14554.755918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14557.962053] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14566.931831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14567.047914] Lustre: 505447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14567.085531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14567.088581] Lustre: Skipped 1 previous similar message [14567.198240] Lustre: lustre-MDT0001: new disk, initializing [14567.310756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14567.348625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14567.367330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14571.338451] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14579.440881] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14585.566674] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14593.938363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14594.220126] Lustre: *** cfs_fail_loc=90f, val=0*** [14594.363852] Lustre: lustre-OST0000: new disk, initializing [14594.368275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14594.387920] Lustre: 507467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14594.560031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14601.206560] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14603.316687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14603.325341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14603.373976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14608.725846] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14618.596476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14618.605593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14618.608314] Lustre: Skipped 1 previous similar message [14618.619505] Lustre: Skipped 1 previous similar message [14621.569142] Lustre: server umount lustre-OST0000 complete [14628.836748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14628.838561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14628.839035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14628.848044] Lustre: Skipped 1 previous similar message [14628.865736] Lustre: Skipped 1 previous similar message [14631.341859] Lustre: server umount lustre-MDT0000 complete [14634.664089] LustreError: 505441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397967 with bad export cookie 16277000077105835167 [14634.670740] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14634.957195] Lustre: server umount lustre-MDT0001 complete [14644.425269] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [14646.952822] Key type lgssc unregistered [14647.269899] LNet: 509369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14647.283821] LNetError: 509369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14647.301408] LNet: Removed LNI 192.168.203.159@tcp [14648.119785] Key type .llcrypt unregistered [14648.121671] Key type ._llcrypt unregistered [14667.343957] Key type ._llcrypt registered [14667.347208] Key type .llcrypt registered [14667.475371] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [14668.390525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14668.506910] alg: No test for adler32 (adler32-zlib) [14669.549648] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [14669.767588] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [14671.455613] Key type lgssc registered [14672.626607] Lustre: Echo OBD driver; http://www.lustre.org/ [14679.861390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14679.880606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14681.341229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14685.410552] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14691.809875] LustreError: 510639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14691.843816] LustreError: 510639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14692.730733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14693.118989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14697.601132] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14704.631411] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14710.153188] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14716.484621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14716.770864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14724.507605] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14733.620416] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14744.041733] loop0: detected capacity change from 0 to 204800000 [14744.210929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14744.219101] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14744.230492] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14748.914890] loop0: detected capacity change from 0 to 204800000 [14749.074563] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14749.082399] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14749.091854] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14753.575228] loop0: detected capacity change from 0 to 204800000 [14758.412380] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14758.578807] Lustre: fs15246-OST0000: new disk, initializing [14758.588596] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14758.605699] Lustre: 514233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14758.660979] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14764.441313] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14773.649685] loop1: detected capacity change from 0 to 204800000 [14779.107672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14779.322747] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14779.389510] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14779.527759] Lustre: fs15246-MDT0000: new disk, initializing [14779.680644] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14779.705605] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14779.854562] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14780.896238] Lustre: 515355:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000891bcc6f x1874222647391616/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787398129 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14784.406416] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14789.987678] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14790.000931] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14790.048072] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14809.061545] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14809.087058] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14812.286975] Lustre: server umount fs15246-MDT0000 complete [14814.187374] LustreError: 510633:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14819.307932] LustreError: 510633:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.189931] Lustre: server umount fs15246-OST0000 complete [14834.657619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14834.665574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14834.676090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14834.696472] Lustre: Skipped 1 previous similar message [14834.706889] Lustre: Skipped 1 previous similar message [14838.589282] Lustre: server umount lustre-OST0000 complete [14844.896543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14844.899036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14844.911258] Lustre: Skipped 1 previous similar message [14844.924235] Lustre: Skipped 1 previous similar message [14848.388147] Lustre: server umount lustre-MDT0000 complete [14850.022386] LustreError: 510639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14850.047080] LustreError: 510639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14852.302400] LustreError: 510618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398185 with bad export cookie 15023252630748972794 [14852.309915] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14852.313893] LustreError: 510618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14852.571569] Lustre: server umount lustre-MDT0001 complete [14863.795841] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [14867.196462] Key type lgssc unregistered [14867.543763] LNet: 518453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14867.573844] LNetError: 518453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14867.602749] LNet: Removed LNI 192.168.203.159@tcp [14868.457973] Key type .llcrypt unregistered [14868.462659] Key type ._llcrypt unregistered [14886.416679] Key type ._llcrypt registered [14886.418870] Key type .llcrypt registered [14886.597966] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [14887.615102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14887.716942] alg: No test for adler32 (adler32-zlib) [14888.848972] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [14889.115554] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [14890.791274] Key type lgssc registered [14892.192449] Lustre: Echo OBD driver; http://www.lustre.org/ [14928.618768] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [14938.345143] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [14971.579660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14971.598645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14972.811231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14972.854810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14972.950297] Lustre: lustre-MDT0000: new disk, initializing [14973.031427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14973.043893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14976.895201] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [14986.387593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14986.462480] Lustre: 522918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14986.505063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14986.508318] Lustre: Skipped 1 previous similar message [14986.589519] Lustre: lustre-MDT0001: new disk, initializing [14986.673917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14986.704932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14986.713927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14991.074827] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [15000.479968] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15006.645791] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15014.257599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15014.562191] Lustre: lustre-OST0000: new disk, initializing [15014.565987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15014.574727] Lustre: 524886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15014.662314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15020.738954] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [15021.659700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15021.669079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15021.764622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15029.317909] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15033.493787] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15033.673483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15037.968808] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15038.163689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15042.532302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15042.534975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15042.544117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15047.651178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15047.662081] Lustre: Skipped 2 previous similar messages [15052.768898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15052.775891] Lustre: Skipped 1 previous similar message [15054.303135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15054.577577] Lustre: server umount lustre-OST0000 complete [15063.008753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15063.029776] Lustre: Skipped 1 previous similar message [15063.035718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15064.965273] Lustre: server umount lustre-MDT0000 complete [15068.834524] LustreError: 522910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398401 with bad export cookie 7193137964015153843 [15068.846595] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15069.131214] Lustre: server umount lustre-MDT0001 complete [15094.752408] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [15106.182247] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [15138.092459] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14844 sec ========= 07:34:28 (1787398468) [15140.041180] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:34:30 (1787398470) === [15143.400129] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:34:34 (1787398474) === [15182.465475] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [15186.029430] Key type lgssc unregistered [15186.390567] LNet: 531534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15186.399978] LNetError: 531534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15186.434652] LNet: Removed LNI 192.168.203.159@tcp [15187.399140] Key type .llcrypt unregistered [15187.402397] Key type ._llcrypt unregistered