[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508237264 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003105] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.014022] *** VALIDATE selinux *** [ 0.022709] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026819] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] *** VALIDATE tmpfs *** [ 0.030499] *** VALIDATE proc *** [ 0.031285] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.034301] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042101] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.045320] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053018] ... event mask: 000000070000000f [ 0.054287] rcu: Hierarchical SRCU implementation. [ 0.056276] smp: Bringing up secondary CPUs ... [ 0.057600] x86: Booting SMP configuration: [ 0.058044] .... node #0, CPUs: #1 #2 #3 [ 0.061408] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145609] node 0 deferred pages initialised in 78ms [ 0.150130] devtmpfs: initialized [ 0.151260] x86/mm: Memory block size: 128MB [ 0.153812] gcov: version magic: 0x41383552 [ 0.155293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157266] pinctrl core: initialized pinctrl subsystem [ 0.158211] [ 0.158794] ************************************************************* [ 0.159018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160014] ** ** [ 0.161014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162015] ** ** [ 0.163017] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165015] ** your system. ** [ 0.166016] ** ** [ 0.167017] ** If you see this message and you are not debugging the ** [ 0.168011] ** kernel, report this immediately to your vendor! ** [ 0.169016] ** ** [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171015] ************************************************************* [ 0.172877] NET: Registered protocol family 16 [ 0.173420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176435] cpuidle: using governor menu [ 0.178000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183761] PCI: Using configuration type 1 for base access [ 0.187139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200037] cryptd: max_cpu_qlen set to 1000 [ 0.202252] ACPI: Added _OSI(Module Device) [ 0.205076] ACPI: Added _OSI(Processor Device) [ 0.206015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221433] ACPI: Interpreter enabled [ 0.223069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225030] ACPI: Using IOAPIC for interrupt routing [ 0.227231] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242886] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257437] acpiphp: Slot [2] registered [ 0.259178] acpiphp: Slot [5] registered [ 0.260166] acpiphp: Slot [6] registered [ 0.262145] acpiphp: Slot [7] registered [ 0.264133] acpiphp: Slot [8] registered [ 0.265128] acpiphp: Slot [9] registered [ 0.267129] acpiphp: Slot [10] registered [ 0.269171] acpiphp: Slot [3] registered [ 0.270119] acpiphp: Slot [4] registered [ 0.272113] acpiphp: Slot [11] registered [ 0.274112] acpiphp: Slot [12] registered [ 0.275113] acpiphp: Slot [13] registered [ 0.277111] acpiphp: Slot [14] registered [ 0.279160] acpiphp: Slot [15] registered [ 0.280124] acpiphp: Slot [16] registered [ 0.282115] acpiphp: Slot [17] registered [ 0.283081] acpiphp: Slot [18] registered [ 0.284109] acpiphp: Slot [19] registered [ 0.286104] acpiphp: Slot [20] registered [ 0.287060] acpiphp: Slot [21] registered [ 0.289101] acpiphp: Slot [22] registered [ 0.290109] acpiphp: Slot [23] registered [ 0.291102] acpiphp: Slot [24] registered [ 0.293160] acpiphp: Slot [25] registered [ 0.294084] acpiphp: Slot [26] registered [ 0.295072] acpiphp: Slot [27] registered [ 0.296000] acpiphp: Slot [28] registered [ 0.296000] acpiphp: Slot [29] registered [ 0.298098] acpiphp: Slot [30] registered [ 0.299092] acpiphp: Slot [31] registered [ 0.300034] PCI host bridge to bus 0000:00 [ 0.301019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320558] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324091] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.333738] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355394] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359954] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363273] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367146] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405616] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454685] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508607] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.566309] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.590018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.629346] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.638017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.646019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.676666] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.684014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724999] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727326] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729426] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.741055] iommu: Default domain type: Passthrough [ 0.742404] SCSI subsystem initialized [ 0.743152] ACPI: bus type USB registered [ 0.744150] usbcore: registered new interface driver usbfs [ 0.746087] usbcore: registered new interface driver hub [ 0.748196] usbcore: registered new device driver usb [ 0.750276] pps_core: LinuxPPS API ver. 1 registered [ 0.752011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756062] PTP clock support registered [ 0.758150] EDAC MC: Ver: 3.0.0 [ 0.759212] PCI: Using ACPI for IRQ routing [ 0.760851] NetLabel: Initializing [ 0.761009] NetLabel: domain hash size = 128 [ 0.762010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763196] NetLabel: unlabeled traffic allowed by default [ 0.766187] vgaarb: loaded [ 0.768335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.778753] clocksource: Switched to clocksource kvm-clock [ 0.893280] VFS: Disk quotas dquot_6.6.0 [ 0.897902] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.900590] *** VALIDATE ramfs *** [ 0.901916] *** VALIDATE hugetlbfs *** [ 0.903709] pnp: PnP ACPI init [ 0.906136] pnp: PnP ACPI: found 6 devices [ 0.930683] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.934345] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937348] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939467] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.942654] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945161] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.948038] NET: Registered protocol family 2 [ 0.951480] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958772] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.963759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972184] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.985992] TCP: Hash tables configured (established 65536 bind 65536) [ 0.991642] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.994590] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997155] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000104] NET: Registered protocol family 1 [ 1.004755] RPC: Registered named UNIX socket transport module. [ 1.007333] RPC: Registered udp transport module. [ 1.009346] RPC: Registered tcp transport module. [ 1.011503] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.014183] NET: Registered protocol family 44 [ 1.016256] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.018827] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021442] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.024325] PCI: CLS 0 bytes, default 64 [ 1.026907] Unpacking initramfs... [ 2.492749] debug: unmapping init [mem 0xffff9557bcc54000-0xffff9557bffbffff] [ 2.496616] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.498821] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.502230] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.012706] Initialise system trusted keyrings [ 3.014599] Key type blacklist registered [ 3.017102] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.027281] zbud: loaded [ 3.031100] *** VALIDATE nfs *** [ 3.032505] *** VALIDATE nfs4 *** [ 3.034260] pstore: using deflate compression [ 3.038465] Platform Keyring initialized [ 3.151959] NET: Registered protocol family 38 [ 3.153417] Key type asymmetric registered [ 3.154615] Asymmetric key parser 'x509' registered [ 3.156034] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.160039] io scheduler mq-deadline registered [ 3.162530] io scheduler kyber registered [ 3.165420] io scheduler bfq registered [ 3.167853] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.170339] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.172989] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.176580] ACPI: Power Button [PWRF] [ 3.182446] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.190342] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.208350] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.216061] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.230832] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.259673] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.289542] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.294724] Non-volatile memory driver v1.3 [ 3.298091] Linux agpgart interface v0.103 [ 3.331554] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.334822] vda: detected capacity change from 0 to 74760192 [ 3.350860] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.352989] vdb: detected capacity change from 0 to 1073741824 [ 3.367177] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.369154] vdc: detected capacity change from 0 to 2621440000 [ 3.384472] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.388256] vdd: detected capacity change from 0 to 2621440000 [ 3.409755] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.412522] vde: detected capacity change from 0 to 4294967296 [ 3.426452] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.429324] vdf: detected capacity change from 0 to 4294967296 [ 3.435344] libphy: Fixed MDIO Bus: probed [ 3.440718] usbcore: registered new interface driver usbserial_generic [ 3.443478] usbserial: USB Serial support registered for generic [ 3.447619] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.453354] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.455271] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.458252] mousedev: PS/2 mouse device common for all mice [ 3.461973] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.463506] rtc_cmos 00:05: RTC can wake from S4 [ 3.470686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.474876] rtc_cmos 00:05: registered as rtc0 [ 3.477876] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.479759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.482149] intel_pstate: CPU model not supported [ 3.490614] hid: raw HID events driver (C) Jiri Kosina [ 3.492911] usbcore: registered new interface driver usbhid [ 3.495307] usbhid: USB HID core driver [ 3.498216] drop_monitor: Initializing network drop monitor service [ 3.501208] Initializing XFRM netlink socket [ 3.503599] NET: Registered protocol family 10 [ 3.507410] Segment Routing with IPv6 [ 3.508943] NET: Registered protocol family 17 [ 3.511811] mpls_gso: MPLS GSO support [ 3.518732] RAS: Correctable Errors collector initialized. [ 3.521437] AVX version of gcm_enc/dec engaged. [ 3.523456] AES CTR mode by8 optimization enabled [ 3.600515] sched_clock: Marking stable (3600491048, 0)->(4597057614, -996566566) [ 3.604766] registered taskstats version 1 [ 3.606972] Loading compiled-in X.509 certificates [ 3.609428] zswap: loaded using pool lzo/zbud [ 3.633412] Key type big_key registered [ 3.645986] Key type encrypted registered [ 3.647905] ima: No TPM chip found, activating TPM-bypass! [ 3.649482] ima: Allocated hash algorithm: sha1 [ 3.651058] ima: No architecture policies found [ 3.653608] evm: Initialising EVM extended attributes: [ 3.655155] evm: security.selinux [ 3.656406] evm: security.ima [ 3.657626] evm: security.capability [ 3.658793] evm: HMAC attrs: 0x1 [ 3.661238] rtc_cmos 00:05: setting system clock to 2026-08-31 05:13:00 UTC (1788153180) [ 3.668201] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.671086] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.679234] Write protecting the kernel read-only data: 28672k [ 3.682238] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.684736] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.716935] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.724559] systemd[1]: Detected virtualization kvm. [ 3.726732] systemd[1]: Detected architecture x86-64. [ 3.730305] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.761656] systemd[1]: No hostname configured. [ 3.763596] systemd[1]: Set hostname to . [ 3.766203] random: systemd: uninitialized urandom read (16 bytes read) [ 3.769596] systemd[1]: Initializing machine ID from random generator. [ 3.815575] random: ln: uninitialized urandom read (6 bytes read) [ 3.916171] random: systemd: uninitialized urandom read (16 bytes read) [ 3.919351] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.924783] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.929441] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.538114] device-mapper: uevent: version 1.0.3 [ 4.543566] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.273140] scsi host0: ata_piix [ 5.287616] virtio_net virtio0 ens2: renamed from eth0 [ 5.297712] scsi host1: ata_piix [ 5.326935] random: fast init done [ 5.355816] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359794] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.601777] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.163927] random: crng init done [ 10.165111] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.587111] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.785234] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.048073] SELinux: Disabled at runtime. [ 12.116352] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.125957] systemd[1]: Detected virtualization kvm. [ 12.128135] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.616192] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.620427] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.628812] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.632982] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.636516] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.645034] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.651384] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ 12.737327] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.093713] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.471872] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.490634] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.558972] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.575711] EDAC sbridge: Ver: 1.1.2 [ 15.465654] Key type dns_resolver registered [ 16.254354] NFS: Registering the id_resolver key type [ 16.261708] Key type id_resolver registered [ 16.264591] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 78.268818] libcfs: loading out-of-tree module taints kernel. [ 78.358292] Key type ._llcrypt registered [ 78.360018] Key type .llcrypt registered [ 78.515797] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 99.116693] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 100.454752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.467787] alg: No test for adler32 (adler32-zlib) [ 102.130357] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 102.432104] hrtimer: interrupt took 4094176 ns [ 103.227738] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 105.065781] Key type lgssc registered [ 108.052080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.521288] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.985422] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 186.664718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 186.737498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.970184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.023434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.113313] Lustre: lustre-MDT0000: new disk, initializing [ 188.207751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.224653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.875818] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.005231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.200644] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.264579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.276271] Lustre: Skipped 1 previous similar message [ 207.413554] Lustre: lustre-MDT0001: new disk, initializing [ 207.506257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.545399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.566136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.265301] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.798692] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.231172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.590219] Lustre: lustre-OST0000: new disk, initializing [ 226.600077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.610138] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.728862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.534671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.549577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.675268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.917895] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.359075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.480790] Lustre: lustre-OST0001: new disk, initializing [ 249.484634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.494717] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.577035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.612555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.617628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.672406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 257.522484] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.785273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.956358] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.478130] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.753730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 295.269438] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 298.102213] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 300.684481] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 302.552311] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:17:57 EDT 2026 [ 319.166824] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 320.582529] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.848901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 335.861563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.877889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.911859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.912522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.922953] Lustre: Skipped 2 previous similar messages [ 341.556110] Lustre: server umount lustre-MDT0000 complete [ 345.193732] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153522 with bad export cookie 9415813837964227355 [ 345.195309] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.200792] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.152145] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.152466] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.158526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.158537] Lustre: Skipped 3 previous similar messages [ 349.183724] Lustre: Skipped 1 previous similar message [ 349.185577] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 350.181410] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.198645] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 351.969111] Lustre: server umount lustre-MDT0001 complete [ 356.802179] Lustre: server umount lustre-OST0000 complete [ 360.935974] Lustre: server umount lustre-OST0001 complete [ 368.918646] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 378.458076] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 416.017319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.277355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.319432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.418231] Lustre: lustre-MDT0000: new disk, initializing [ 416.501365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.524237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.365659] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 429.997114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.150734] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.208550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.212539] Lustre: Skipped 1 previous similar message [ 430.345941] Lustre: lustre-MDT0001: new disk, initializing [ 430.489658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.531621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.566927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.197847] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 445.973257] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.507474] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.911319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.457057] Lustre: lustre-OST0000: new disk, initializing [ 461.465977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.472996] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.601314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.395573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.405255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.591118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.818313] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 478.152744] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 482.483737] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 482.616844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.841465] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 487.091925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 489.443813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.446304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.453291] LustreError: Skipped 1 previous similar message [ 489.454323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.469669] Lustre: Skipped 1 previous similar message [ 489.483637] Lustre: Skipped 4 previous similar messages [ 494.561091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.571314] Lustre: Skipped 1 previous similar message [ 503.263151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 503.379078] Lustre: server umount lustre-OST0000 complete [ 507.373426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.384499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.401651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.405400] Lustre: Skipped 2 previous similar messages [ 513.291672] Lustre: server umount lustre-MDT0000 complete [ 515.044546] LustreError: 15892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.065643] LustreError: 15892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 516.741192] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153693 with bad export cookie 9415813837964228888 [ 516.755251] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.760608] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 517.122913] Lustre: server umount lustre-MDT0001 complete [ 520.857521] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:21:35 (1788153695) [ 523.047636] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 524.608405] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:21:39 (1788153699) [ 530.807334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.292116] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.434273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.546513] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 536.547052] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.102781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.700467] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 556.660303] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.184744] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 568.639286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.945392] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 582.839517] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 590.818717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 590.836601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.847018] Lustre: Skipped 1 previous similar message [ 590.851319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 590.855459] Lustre: Skipped 2 previous similar messages [ 595.919658] Lustre: server umount lustre-OST0000 complete [ 601.061777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.074237] LustreError: Skipped 1 previous similar message [ 605.540083] Lustre: server umount lustre-MDT0000 complete [ 607.203703] LustreError: 20700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.227274] LustreError: 20700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 608.993016] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153785 with bad export cookie 9415813837964229770 [ 608.997727] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.010545] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 609.301855] Lustre: server umount lustre-MDT0001 complete [ 618.934591] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 620.991379] Key type lgssc unregistered [ 621.173926] LNet: 22990:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.184062] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.196828] LNet: Removed LNI 192.168.204.120@tcp [ 621.772459] Key type .llcrypt unregistered [ 621.776415] Key type ._llcrypt unregistered [ 642.673091] Key type ._llcrypt registered [ 642.676023] Key type .llcrypt registered [ 642.815683] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 643.910606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.998204] alg: No test for adler32 (adler32-zlib) [ 645.037592] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 645.295786] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 646.967131] Key type lgssc registered [ 648.106731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.100696] loop: module loaded [ 655.141739] loop0: detected capacity change from 0 to 204800000 [ 655.311887] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.317859] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.599759] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.602169] loop0: detected capacity change from 0 to 204800000 [ 660.770950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.785978] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.802785] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 664.915509] loop0: detected capacity change from 0 to 204800000 [ 668.730975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.749119] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.965796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 670.025451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 670.094397] Lustre: lustre-MDT0000: new disk, initializing [ 670.173960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.187052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 673.684961] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 682.673099] loop1: detected capacity change from 0 to 204800000 [ 688.465132] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.757868] Lustre: lustre-OST07c6: new disk, initializing [ 688.760884] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 688.764339] Lustre: Skipped 1 previous similar message [ 688.769077] Lustre: 26176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 688.841583] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 694.424542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 694.430093] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 694.462349] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 695.271551] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 700.025442] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 700.039054] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 700.054305] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 700.064492] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 700.076949] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 700.086429] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.295658] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.302632] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.308887] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.323729] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.333475] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.336984] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.736249] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 722.750243] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.761670] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.767396] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.773533] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.783297] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.792724] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 722.799863] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.804625] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 722.814214] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.820619] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.833965] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.024652] Lustre: 26466:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 726.029896] Lustre: 26466:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 726.036417] Lustre: 26466:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 726.042641] Lustre: 26466:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.046240] Lustre: 26466:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 726.050502] Lustre: 26466:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.058242] Lustre: 26466:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 726.065709] Lustre: 26466:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.070123] Lustre: 26466:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 726.073170] Lustre: 26466:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.076902] Lustre: 26466:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.081973] Lustre: 26466:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.303458] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 731.326519] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 731.339536] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 731.353469] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.359349] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 731.365792] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.369147] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 731.373854] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.378286] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 731.388125] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.392471] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.398741] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.503722] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 742.510456] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 742.520571] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.526826] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.537148] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.543804] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.548197] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 742.553252] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.557215] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.561979] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.566873] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.573416] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.521282] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.532649] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 796.644647] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 801.760754] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 804.319146] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 804.607606] Lustre: server umount lustre-OST07c6 complete [ 815.448171] Lustre: server umount lustre-MDT0000 complete [ 834.590851] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 837.512576] Key type lgssc unregistered [ 837.857035] LNet: 28736:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.863910] LNetError: 28736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.885187] LNet: Removed LNI 192.168.204.120@tcp [ 838.846968] Key type .llcrypt unregistered [ 838.850504] Key type ._llcrypt unregistered [ 859.235934] Key type ._llcrypt registered [ 859.242983] Key type .llcrypt registered [ 859.380392] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 860.629485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.660798] alg: No test for adler32 (adler32-zlib) [ 861.917213] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 862.096546] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 863.711242] Key type lgssc registered [ 864.925626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 873.410760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 873.430805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.903964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.224564] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 884.203467] LustreError: 30007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.239067] LustreError: 30007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 888.065795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.562376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.065969] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 902.040350] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.332846] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.048622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 917.453537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.543418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 923.726109] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 931.395408] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.221388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.231785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.241137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.259916] Lustre: Skipped 1 previous similar message [ 950.117695] Lustre: server umount lustre-OST0000 complete [ 954.340723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.344067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.361211] Lustre: Skipped 1 previous similar message [ 954.367676] Lustre: Skipped 1 previous similar message [ 959.458497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.462777] Lustre: Skipped 2 previous similar messages [ 964.577625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.584313] Lustre: Skipped 1 previous similar message [ 968.159180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 968.392773] Lustre: server umount lustre-MDT0000 complete [ 969.698715] LustreError: 30007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.215312] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154149 with bad export cookie 7561347746037019141 [ 972.229146] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.230857] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 972.536890] Lustre: server umount lustre-MDT0001 complete [ 983.626316] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 986.047313] Key type lgssc unregistered [ 986.224809] LNet: 33744:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.228305] LNetError: 33744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.239632] LNet: Removed LNI 192.168.204.120@tcp [ 986.887235] Key type .llcrypt unregistered [ 986.888564] Key type ._llcrypt unregistered [ 1006.869404] Key type ._llcrypt registered [ 1006.872702] Key type .llcrypt registered [ 1006.946959] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1008.103898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.259277] alg: No test for adler32 (adler32-zlib) [ 1009.857635] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1010.013148] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1011.681951] Key type lgssc registered [ 1013.180185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.738749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.757850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.299039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.304473] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1033.703566] LustreError: 35014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.730900] LustreError: 35014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1037.431898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.941831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.735497] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1049.955495] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1055.664437] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.360179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.080565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.166072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1072.805289] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1081.897512] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.088469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.099924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.103626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.121845] Lustre: Skipped 1 previous similar message [ 1098.231451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.243306] Lustre: Skipped 2 previous similar messages [ 1099.072312] Lustre: server umount lustre-OST0000 complete [ 1108.453401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.458632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.478635] Lustre: Skipped 1 previous similar message [ 1108.485213] Lustre: Skipped 1 previous similar message [ 1111.337099] Lustre: server umount lustre-MDT0000 complete [ 1113.571238] LustreError: 35743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.602789] LustreError: 35743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1115.772846] LustreError: 37796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154292 with bad export cookie 1441541542065747759 [ 1115.780895] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1115.785651] LustreError: 37796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1116.427835] Lustre: server umount lustre-MDT0001 complete [ 1130.567739] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1133.713985] Key type lgssc unregistered [ 1134.056397] LNet: 38548:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.070418] LNetError: 38548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.100254] LNet: Removed LNI 192.168.204.120@tcp [ 1134.912759] Key type .llcrypt unregistered [ 1134.918147] Key type ._llcrypt unregistered [ 1161.111937] Key type ._llcrypt registered [ 1161.116959] Key type .llcrypt registered [ 1161.311518] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1162.032960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.055897] alg: No test for adler32 (adler32-zlib) [ 1163.436808] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1163.719406] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1165.449158] Key type lgssc registered [ 1166.967400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.975620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.992664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.794303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.594545] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1187.325758] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.344168] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1191.871809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.215390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.232463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1206.389463] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.557493] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.175797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.813292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.841118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1229.573986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1238.159061] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.993549] LustreError: 41164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.213656] Lustre: server umount lustre-MDT0000 complete [ 1247.722484] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.729365] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.749973] Lustre: Skipped 2 previous similar messages [ 1247.781284] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1249.671076] LustreError: 39949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154426 with bad export cookie 8214974586829728057 [ 1249.672777] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.682605] LustreError: 39949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1250.085827] LustreError: 39962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.144847] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 1252.834360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.850674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1254.882836] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.883610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1254.912797] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1256.098994] Lustre: server umount lustre-MDT0001 complete [ 1276.778340] Lustre: server umount lustre-OST0000 complete [ 1296.718514] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1299.283416] Key type lgssc unregistered [ 1299.658926] LNet: 43829:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.665227] LNetError: 43829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.678595] LNet: Removed LNI 192.168.204.120@tcp [ 1300.679771] Key type .llcrypt unregistered [ 1300.681677] Key type ._llcrypt unregistered [ 1327.122532] Key type ._llcrypt registered [ 1327.124971] Key type .llcrypt registered [ 1327.258954] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1328.404873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.616595] alg: No test for adler32 (adler32-zlib) [ 1329.817321] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1330.049787] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1331.712164] Key type lgssc registered [ 1333.526808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.613813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.632761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.146598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.111737] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1353.700106] LustreError: 45259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.722901] LustreError: 45259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1358.797317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.832371] LustreError: 45260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.222756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.848791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1371.555966] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1377.162892] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1383.646855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.913631] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1384.041481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.095989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1391.802264] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1401.951897] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.515710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1409.529756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.549698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.531824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1410.545965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.577903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.582085] Lustre: Skipped 2 previous similar messages [ 1415.447236] Lustre: server umount lustre-OST0000 complete [ 1436.001598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.026177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.035727] Lustre: Skipped 1 previous similar message [ 1440.230576] Lustre: server umount lustre-MDT0000 complete [ 1441.255154] LustreError: 45988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.285677] LustreError: 45988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1444.244863] LustreError: 45241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154621 with bad export cookie 14665417228275033870 [ 1444.247914] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.260148] LustreError: 45241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1450.500478] Lustre: server umount lustre-MDT0001 complete [ 1460.988807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1463.628383] Key type lgssc unregistered [ 1463.937698] LNet: 48971:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.954414] LNetError: 48971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.989816] LNet: Removed LNI 192.168.204.120@tcp [ 1464.836305] Key type .llcrypt unregistered [ 1464.841140] Key type ._llcrypt unregistered [ 1490.979214] Key type ._llcrypt registered [ 1490.981927] Key type .llcrypt registered [ 1491.091598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1492.526076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1492.542962] alg: No test for adler32 (adler32-zlib) [ 1493.614849] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1493.885388] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1495.567568] Key type lgssc registered [ 1497.001747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.052906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1505.074571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.629363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1511.483429] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1517.025253] LustreError: 50388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.053830] LustreError: 50388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1521.380296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.745014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1527.111184] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1535.058398] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1540.869833] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1547.851362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.102403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.222152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1554.962849] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1562.840797] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1571.146937] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1574.449328] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1586.681847] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h39m20s [ 1588.119117] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1590.075854] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1591.140270] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1592.290704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.296338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.300547] Lustre: Skipped 1 previous similar message [ 1592.313391] Lustre: Skipped 2 previous similar messages [ 1596.191743] LustreError: 50384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.308557] Lustre: server umount lustre-MDT0000 complete [ 1599.383121] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1599.463639] LustreError: 50385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.482960] LustreError: 50385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1602.344632] LustreError: 50385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.835946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.992503] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.205302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.289213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.223713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.677295] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1614.400995] Lustre: DEBUG MARKER: Wait for df (36669) ... [ 1615.975812] Lustre: DEBUG MARKER: done [ 1621.478171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1621.493428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.516942] Lustre: Skipped 1 previous similar message [ 1621.521919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.525283] Lustre: Skipped 3 previous similar messages [ 1622.498400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.621858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.633317] Lustre: Skipped 3 previous similar messages [ 1634.271122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1634.494340] Lustre: server umount lustre-OST0000 complete [ 1642.984543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.985241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1642.993170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.993178] Lustre: Skipped 2 previous similar messages [ 1643.000435] Lustre: Skipped 1 previous similar message [ 1644.469639] Lustre: server umount lustre-MDT0000 complete [ 1648.360475] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154825 with bad export cookie 16823154840032931985 [ 1648.364159] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1648.371744] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1648.774515] Lustre: server umount lustre-MDT0001 complete [ 1660.407907] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1663.079320] Key type lgssc unregistered [ 1663.412193] LNet: 55314:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.421290] LNetError: 55314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.447981] LNet: Removed LNI 192.168.204.120@tcp [ 1664.252473] Key type .llcrypt unregistered [ 1664.254662] Key type ._llcrypt unregistered [ 1705.332223] Key type ._llcrypt registered [ 1705.334955] Key type .llcrypt registered [ 1705.423693] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1706.428028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.515978] alg: No test for adler32 (adler32-zlib) [ 1707.620499] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1707.920185] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1709.608129] Key type lgssc registered [ 1710.673612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.812049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.851039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.180961] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1720.202329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1720.388541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.167105] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1733.483242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.554373] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1733.577830] Lustre: 57521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1733.774510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.290185] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1746.958519] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.795561] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1761.068260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.382427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1761.630241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.816902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1770.216250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1777.168535] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1783.716951] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h42m38s [ 1785.526651] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1793.387846] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1796.464622] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1796.731947] LustreError: 58938:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1816.937106] Lustre: lustre-MDT0000: Client 1f90dbbd-acf5-41e5-b96a-1ba427f3592c (at 192.168.204.20@tcp) reconnecting [ 1836.792462] LustreError: 58938:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1839.042256] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1841.637982] Lustre: DEBUG MARKER: done [ 1851.883031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1851.903062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.931649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1852.385078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1852.403107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.154722] Lustre: server umount lustre-OST0000 complete [ 1862.112554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.125141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.139037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.142996] Lustre: Skipped 2 previous similar messages [ 1863.202906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.212057] Lustre: Skipped 2 previous similar messages [ 1864.746282] Lustre: server umount lustre-MDT0000 complete [ 1868.689531] LustreError: 59385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155045 with bad export cookie 1415591555020604433 [ 1868.696330] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.703335] LustreError: 59385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1868.996583] Lustre: server umount lustre-MDT0001 complete [ 1879.883260] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1883.350612] Key type lgssc unregistered [ 1883.718180] LNet: 61855:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.736403] LNetError: 61855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.776482] LNet: Removed LNI 192.168.204.120@tcp [ 1885.744602] Key type .llcrypt unregistered [ 1885.748972] Key type ._llcrypt unregistered [ 1935.794716] Key type ._llcrypt registered [ 1935.799211] Key type .llcrypt registered [ 1935.953172] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1937.035647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.212796] alg: No test for adler32 (adler32-zlib) [ 1938.478143] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1938.874901] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1940.583283] Key type lgssc registered [ 1941.935269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1945.943115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1945.961654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.313334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1947.333323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.626441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.703818] Lustre: Failing over lustre-MDT0000 [ 1949.125924] Lustre: server umount lustre-MDT0000 complete [ 1956.969955] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:45:31 (1788155131) [ 1963.641766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.106435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.662989] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1977.069679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.220077] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1977.242484] Lustre: 64657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.553782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.048477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1989.986451] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1996.188143] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2002.730631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2002.924060] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2003.178354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.055981] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2010.355232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2018.385530] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.215479] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2034.144456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2034.152929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.173262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.199288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.199807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.222488] Lustre: Skipped 2 previous similar messages [ 2038.058493] Lustre: server umount lustre-MDT0000 complete [ 2041.874443] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155218 with bad export cookie 17614530672583783860 [ 2041.880948] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.884952] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.314855] Lustre: server umount lustre-MDT0001 complete [ 2046.071419] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2050.749833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2060.495133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.817359] Lustre: 63579:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155223/real 1788155223] req@00000000c4162100 x1875016537908608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2062.862954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.883246] Lustre: Skipped 1 previous similar message [ 2067.391424] LustreError: 66505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.414248] LustreError: 66505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2067.527934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.644555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2068.513232] Lustre: 63576:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155228/real 1788155228] req@00000000cbb3d594 x1875016537908736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.521051] LustreError: 66506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.054671] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2072.545741] Lustre: 63577:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155233/real 1788155233] req@00000000f033d6f5 x1875016537908992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155249 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.549812] LustreError: 66505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.585663] LustreError: 66505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2077.665268] Lustre: 63579:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155238/real 1788155238] req@000000000407c584 x1875016537909120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155254 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.668608] LustreError: 68425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.717100] LustreError: 68425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2080.352659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.921670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.924899] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2086.387585] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2091.555588] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2093.083926] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.853948] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2105.312973] Lustre: server umount lustre-MDT0000 complete [ 2106.343058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2106.361692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.385424] LustreError: 66506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.430674] LustreError: 67366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155286 with bad export cookie 17614530672583786142 [ 2109.433167] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.459122] LustreError: 67366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2109.791245] Lustre: server umount lustre-MDT0001 complete [ 2116.112445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2126.062405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.327586] Lustre: 63578:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155288/real 1788155288] req@00000000eeb223d0 x1875016537933568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2127.359134] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.371674] Lustre: Skipped 2 previous similar messages [ 2134.565682] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf473625228944401 [ 2134.579728] Lustre: MGC192.168.204.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2134.785993] LustreError: 69161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.808677] LustreError: 69161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2134.856016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.927922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2135.851135] Lustre: 63578:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155296/real 1788155296] req@0000000005c311b6 x1875016537934080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.866743] Lustre: 63578:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2138.875842] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2146.290765] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2148.684996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.030700] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2154.471119] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2162.259955] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.769293] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2180.065805] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.066469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.068198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2180.084627] Lustre: Skipped 2 previous similar messages [ 2180.103821] Lustre: Skipped 2 previous similar messages [ 2183.521445] Lustre: server umount lustre-MDT0000 complete [ 2185.189973] LustreError: 71110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.215793] LustreError: 71110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2186.854579] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155363 with bad export cookie 17614530672583787521 [ 2186.855927] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.873586] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2187.108807] Lustre: server umount lustre-MDT0001 complete [ 2193.667092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.368580] Lustre: DEBUG MARKER: files compared the same [ 2209.771913] Lustre: server umount lustre-OST0000 complete [ 2226.716437] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2228.957496] Key type lgssc unregistered [ 2229.186429] LNet: 74387:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.192430] LNetError: 74387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.225096] LNet: Removed LNI 192.168.204.120@tcp [ 2229.916772] Key type .llcrypt unregistered [ 2229.918934] Key type ._llcrypt unregistered [ 2254.402299] Key type ._llcrypt registered [ 2254.411687] Key type .llcrypt registered [ 2254.544807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2255.336931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.479485] alg: No test for adler32 (adler32-zlib) [ 2256.519959] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2256.759795] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2258.401095] Key type lgssc registered [ 2259.797542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2268.124420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2268.147105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.600060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.801228] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2279.905191] LustreError: 75805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.922040] LustreError: 75805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2281.970161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.260761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2286.703740] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2294.421889] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2299.715074] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.224894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.517957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.663745] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2314.732254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2320.551340] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2330.080495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.083606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2330.083776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.102036] Lustre: Skipped 1 previous similar message [ 2335.199920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.210990] Lustre: Skipped 1 previous similar message [ 2340.322672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.335237] Lustre: Skipped 1 previous similar message [ 2343.391143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2343.496580] Lustre: server umount lustre-OST0000 complete [ 2349.025071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.038956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.052018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.057484] Lustre: Skipped 1 previous similar message [ 2350.562852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.683495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.690449] Lustre: Skipped 3 previous similar messages [ 2361.312291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.589733] Lustre: server umount lustre-MDT0000 complete [ 2365.182331] LustreError: 75786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155542 with bad export cookie 1651367915743162153 [ 2365.200841] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.470570] Lustre: server umount lustre-MDT0001 complete [ 2375.360141] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2377.856421] Key type lgssc unregistered [ 2378.128932] LNet: 79337:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.139327] LNetError: 79337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.156258] LNet: Removed LNI 192.168.204.120@tcp [ 2379.002418] Key type .llcrypt unregistered [ 2379.004755] Key type ._llcrypt unregistered [ 2403.064957] Key type ._llcrypt registered [ 2403.071545] Key type .llcrypt registered [ 2403.160721] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2403.977704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.274067] alg: No test for adler32 (adler32-zlib) [ 2405.377939] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2405.670561] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2407.415251] Key type lgssc registered [ 2408.575893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.080499] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2415.093668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.559493] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2435.679519] LustreError: 80729:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 2462.367240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.897304] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2475.604593] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.325057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.432119] Lustre: *** cfs_fail_loc=706, val=0*** [ 2482.802770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.912702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2486.929637] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2488.295716] LustreError: 80755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.309079] LustreError: 80755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2493.429039] LustreError: 80754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.492809] LustreError: 82509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.525272] LustreError: 82509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2497.254946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.887923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.472514] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2510.663677] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2517.537945] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2523.624224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.644030] Lustre: Skipped 1 previous similar message [ 2523.649847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.655156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2523.662324] LustreError: Skipped 1 previous similar message [ 2528.741318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.750753] Lustre: Skipped 1 previous similar message [ 2533.861022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2533.872613] Lustre: Skipped 1 previous similar message [ 2534.367148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2534.490873] Lustre: server umount lustre-OST0000 complete [ 2538.987260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.987458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2538.995594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.012055] Lustre: Skipped 2 previous similar messages [ 2544.098751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.667803] Lustre: server umount lustre-MDT0000 complete [ 2548.431277] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155725 with bad export cookie 11040134170132492318 [ 2548.441994] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.449754] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2548.799227] Lustre: server umount lustre-MDT0001 complete [ 2560.945073] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2563.680519] Key type lgssc unregistered [ 2563.956663] LNet: 84346:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.963674] LNetError: 84346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.984523] LNet: Removed LNI 192.168.204.120@tcp [ 2564.702743] Key type .llcrypt unregistered [ 2564.705405] Key type ._llcrypt unregistered [ 2593.864344] Key type ._llcrypt registered [ 2593.865984] Key type .llcrypt registered [ 2593.965379] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2595.345585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2595.387196] alg: No test for adler32 (adler32-zlib) [ 2596.550705] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2596.763246] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2598.463329] Key type lgssc registered [ 2599.512891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2607.189478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2607.225586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.427230] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2619.237679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.762097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2624.056901] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2624.992378] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.015907] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2630.118567] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.144114] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2632.626950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.993676] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.014497] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2633.030813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.306280] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.336609] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2640.114216] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2643.428182] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.448479] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2648.488250] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.501836] Lustre: MGS: Client 3d4c0254-d2e5-42a4-839c-d7a2c0c09fdb (at 0@lo) reconnecting [ 2648.518096] LustreError: 86466:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2648.696185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.776409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2653.014931] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2669.025729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.035255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.058778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.543273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.320484] Lustre: server umount lustre-OST0000 complete [ 2679.777836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.790110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.799651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.813152] Lustre: Skipped 1 previous similar message [ 2679.818207] Lustre: Skipped 1 previous similar message [ 2680.892485] Lustre: server umount lustre-MDT0000 complete [ 2685.028685] Lustre: server umount lustre-MDT0001 complete [ 2688.977185] Lustre: server umount MGS complete [ 2700.813210] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:57:55 (1788155875) [ 2731.971899] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 2741.508150] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2778.532623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.369767] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2794.781169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.005996] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2795.102898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2795.287933] Lustre: lustre-MDT0001: new disk, initializing [ 2795.384827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.409403] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2795.419149] Lustre: 93622:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2795.426509] LustreError: 93622:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2800.100398] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2811.361996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.720181] Lustre: lustre-OST0000: new disk, initializing [ 2811.732262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2811.745244] Lustre: 94388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2811.841913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.572186] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2821.732249] Lustre: 94844:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ea37631a x1875017227870080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788156014 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2829.413727] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2836.355098] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.366250] Lustre: MGS: Client 008a60ec-3e44-421a-bbd2-45db27a112a5 (at 0@lo) reconnecting [ 2836.415918] Lustre: 92890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2836.430442] LustreError: 93608:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2836.439456] LustreError: 93608:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2836.473214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2836.603478] Lustre: lustre-MDT0000: new disk, initializing [ 2836.739187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.782476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2836.809239] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2837.855967] Lustre: 95458:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000021bf7d6 x1875017227877120/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788156030 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2841.458147] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2842.985210] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2846.954753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2846.970802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2847.053391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2872.802950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2872.814579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.840713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.851059] Lustre: Skipped 1 previous similar message [ 2878.390122] Lustre: server umount lustre-OST0000 complete [ 2884.585248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.587641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.601914] Lustre: Skipped 2 previous similar messages [ 2884.622235] Lustre: Skipped 2 previous similar messages [ 2888.452151] Lustre: server umount lustre-MDT0000 complete [ 2889.695786] LustreError: 93592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.715775] LustreError: 93592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2898.925253] Lustre: server umount lustre-MDT0001 complete [ 2902.970552] Lustre: server umount MGS complete [ 2913.588525] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:01:28 (1788156088) [ 2930.833888] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2933.292693] Key type lgssc unregistered [ 2933.647570] LNet: 97969:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.653496] LNetError: 97969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.674312] LNet: Removed LNI 192.168.204.120@tcp [ 2934.581729] Key type .llcrypt unregistered [ 2934.585620] Key type ._llcrypt unregistered [ 2955.944682] Key type ._llcrypt registered [ 2955.948464] Key type .llcrypt registered [ 2956.056958] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2957.087972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2957.161073] alg: No test for adler32 (adler32-zlib) [ 2958.237082] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2958.608312] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2960.383397] Key type lgssc registered [ 2961.887498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2970.098804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2970.132262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.528113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.082165] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2981.860632] LustreError: 99190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.880640] LustreError: 99190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2985.329150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.871855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.775608] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2998.926563] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3004.252874] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3010.914841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.217096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.372651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3017.401725] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3026.301488] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3041.261793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.277322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3041.279763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.662483] Lustre: server umount lustre-OST0000 complete [ 3051.495539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.496340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3051.516960] Lustre: Skipped 2 previous similar messages [ 3051.522346] Lustre: Skipped 1 previous similar message [ 3055.834576] Lustre: server umount lustre-MDT0000 complete [ 3056.608704] LustreError: 99190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.627908] LustreError: 99190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3059.903399] LustreError: 99172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156236 with bad export cookie 8213217235709216859 [ 3059.913940] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.920808] LustreError: 99172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3060.243803] Lustre: server umount lustre-MDT0001 complete [ 3071.404397] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3074.204766] Key type lgssc unregistered [ 3074.552566] LNet: 102747:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3074.562684] LNetError: 102747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3074.605190] LNet: Removed LNI 192.168.204.120@tcp [ 3075.581450] Key type .llcrypt unregistered [ 3075.584693] Key type ._llcrypt unregistered [ 3092.937975] Key type ._llcrypt registered [ 3092.941225] Key type .llcrypt registered [ 3093.045341] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3094.138910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3094.188686] alg: No test for adler32 (adler32-zlib) [ 3095.187709] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3095.427701] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3097.112066] Key type lgssc registered [ 3098.371137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.393195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.403745] LustreError: 103825:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3104.452349] LustreError: 103832:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3104.459404] LustreError: 103825:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4955 [ 3104.468357] LustreError: 103825:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3104.472663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.477276] LustreError: 103825:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3106.212456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.402293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.875101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.904140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3124.757522] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lsmod [ 3127.946916] LustreError: 105039:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3127.955574] LustreError: 105040:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3127.961979] LustreError: 105039:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3127.969679] LustreError: 105039:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3127.971755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.974983] LustreError: 105039:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3128.474396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.545956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3135.457982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.463382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.472484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.974651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.981586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.982956] Lustre: Skipped 1 previous similar message [ 3135.994553] Lustre: Skipped 1 previous similar message [ 3137.753734] Lustre: server umount lustre-MDT0000 complete [ 3140.068978] LustreError: 103866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.084668] LustreError: 103866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3141.345348] LustreError: 103848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156318 with bad export cookie 8356739595411697809 [ 3141.346337] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.355571] LustreError: 103848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3142.175633] LustreError: 104302:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000031f77f37 x1875017751089152/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3142.302475] Lustre: server umount lustre-MDT0001 complete [ 3156.104921] Lustre: server umount lustre-OST0000 complete [ 3163.226536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.616363] LustreError: 106104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.694291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.135106] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3168.742560] LustreError: 106105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.859945] LustreError: 106104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.108468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.533592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3180.978763] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3188.955154] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3195.442092] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3203.720098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.252055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.370785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3211.279514] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3219.872758] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.108592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.127685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.129329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.235256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.241977] Lustre: Skipped 2 previous similar messages [ 3233.271104] Lustre: server umount lustre-OST0000 complete [ 3237.344877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.365963] Lustre: Skipped 1 previous similar message [ 3237.369790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.376631] Lustre: Skipped 1 previous similar message [ 3243.464741] Lustre: server umount lustre-MDT0000 complete [ 3246.784430] LustreError: 106086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156423 with bad export cookie 8356739595411698719 [ 3246.797790] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3246.808517] LustreError: 106086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3247.140679] Lustre: server umount lustre-MDT0001 complete [ 3259.463886] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3262.346845] Key type lgssc unregistered [ 3262.599984] LNet: 109639:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3262.608872] LNetError: 109639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3262.622842] LNet: Removed LNI 192.168.204.120@tcp [ 3263.438853] Key type .llcrypt unregistered [ 3263.441224] Key type ._llcrypt unregistered [ 3287.277488] Key type ._llcrypt registered [ 3287.280346] Key type .llcrypt registered [ 3287.376423] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3288.083893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3288.121447] alg: No test for adler32 (adler32-zlib) [ 3289.208386] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3289.389832] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3291.047195] Key type lgssc registered [ 3292.064712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3298.937265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3298.954658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.377428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.120257] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3310.565511] LustreError: 111069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.593889] LustreError: 111069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3311.979914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.358918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3317.160109] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3325.772501] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.582172] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3339.435828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.779360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.989139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3346.643973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3353.614818] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3363.399190] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3367.341580] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3372.515570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.516566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3372.525149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.634746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.654311] Lustre: Skipped 2 previous similar messages [ 3378.741816] Lustre: server umount lustre-OST0000 complete [ 3382.752698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.767510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.770677] Lustre: Skipped 2 previous similar messages [ 3382.790435] Lustre: Skipped 1 previous similar message [ 3387.873308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.881717] Lustre: Skipped 1 previous similar message [ 3388.878083] Lustre: server umount lustre-MDT0000 complete [ 3392.839797] LustreError: 111050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156569 with bad export cookie 7951507096664968502 [ 3392.851638] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.103623] Lustre: server umount lustre-MDT0001 complete [ 3405.124477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3407.858279] Key type lgssc unregistered [ 3408.092501] LNet: 114710:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.097909] LNetError: 114710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3408.120504] LNet: Removed LNI 192.168.204.120@tcp [ 3409.018851] Key type .llcrypt unregistered [ 3409.022138] Key type ._llcrypt unregistered [ 3426.909755] Key type ._llcrypt registered [ 3426.912324] Key type .llcrypt registered [ 3427.028768] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3428.291446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3428.343903] alg: No test for adler32 (adler32-zlib) [ 3429.486564] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3429.751882] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3431.440319] Key type lgssc registered [ 3432.689895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3439.766835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3439.795746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.250630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.477436] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3451.885662] LustreError: 115934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.923878] LustreError: 115934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3454.742607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.289864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.110246] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3468.063161] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3474.274141] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3479.951324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.252025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.409330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3486.552386] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3495.140833] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3507.168118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.179693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.182860] LustreError: Skipped 1 previous similar message [ 3507.183731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.191521] Lustre: Skipped 1 previous similar message [ 3507.206187] Lustre: Skipped 1 previous similar message [ 3509.965584] Lustre: server umount lustre-OST0000 complete [ 3516.896059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3516.901402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.913200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.476229] Lustre: server umount lustre-MDT0000 complete [ 3522.529189] LustreError: 115930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.551571] LustreError: 115930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3522.967911] LustreError: 115915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156699 with bad export cookie 17140733506612520525 [ 3522.980788] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.984571] LustreError: 115915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3523.409408] Lustre: server umount lustre-MDT0001 complete [ 3536.595633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3539.764746] Key type lgssc unregistered [ 3540.023526] LNet: 119467:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3540.034935] LNetError: 119467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.060337] LNet: Removed LNI 192.168.204.120@tcp [ 3541.099228] Key type .llcrypt unregistered [ 3541.102611] Key type ._llcrypt unregistered [ 3573.208518] Key type ._llcrypt registered [ 3573.210869] Key type .llcrypt registered [ 3573.327019] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3574.316673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3574.367743] alg: No test for adler32 (adler32-zlib) [ 3575.611397] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3576.058721] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3577.855190] Key type lgssc registered [ 3579.255301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3587.999836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3588.024124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.626526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.886169] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3599.850401] LustreError: 120885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.869863] LustreError: 120885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3602.034673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.397798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3605.885684] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3614.034059] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.427771] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3628.109969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.792927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.927886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3636.899445] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3644.696336] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3654.764957] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3658.820796] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3658.827805] Lustre: Skipped 2 previous similar messages [ 3663.269530] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3663.274616] Lustre: Skipped 2 previous similar messages [ 3664.440782] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3664.474704] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3672.805738] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3672.818578] Lustre: Skipped 2 previous similar messages [ 3681.349826] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp [ 3686.846632] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.20/24@tcp in log lustre-MDT0000 [ 3686.853621] Lustre: Skipped 2 previous similar messages [ 3689.082093] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 3689.086021] Lustre: Skipped 1 previous similar message [ 3699.179998] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3699.180690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.189111] LustreError: Skipped 1 previous similar message [ 3699.228328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.785602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.790671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.797928] LustreError: Skipped 3 previous similar messages [ 3703.808654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.060553] Lustre: server umount lustre-OST0000 complete [ 3708.897258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.899238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3708.908639] Lustre: Skipped 1 previous similar message [ 3708.914781] Lustre: Skipped 2 previous similar messages [ 3714.019960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.028738] Lustre: Skipped 1 previous similar message [ 3719.136921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.144078] Lustre: Skipped 1 previous similar message [ 3723.231200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3723.405304] Lustre: server umount lustre-MDT0000 complete [ 3727.500470] LustreError: 120864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156904 with bad export cookie 4508667504196333311 [ 3727.505583] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.512206] LustreError: 120864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3727.749348] Lustre: server umount lustre-MDT0001 complete [ 3740.007651] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3743.595587] Key type lgssc unregistered [ 3743.972925] LNet: 125770:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3743.982145] LNetError: 125770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3744.032850] LNet: Removed LNI 192.168.204.120@tcp [ 3745.212592] Key type .llcrypt unregistered [ 3745.215145] Key type ._llcrypt unregistered [ 3781.302665] Key type ._llcrypt registered [ 3781.307927] Key type .llcrypt registered [ 3781.481443] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3782.870916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3782.902806] alg: No test for adler32 (adler32-zlib) [ 3784.032466] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3784.273076] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3785.999611] Key type lgssc registered [ 3787.887616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.208849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3798.224291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.918135] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3799.943890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.953893] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.20/24@tcp [ 3804.516220] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3809.319318] LustreError: 127336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.335841] LustreError: 127336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3814.383122] LustreError: 127335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.284834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.830373] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3815.877165] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 3815.901229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.661553] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3831.625718] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3838.285843] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3846.449510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.786173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3854.262400] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3863.347830] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.315885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3885.038356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.041478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.057842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.069636] Lustre: Skipped 1 previous similar message [ 3885.083950] Lustre: Skipped 1 previous similar message [ 3889.000355] Lustre: server umount lustre-OST0000 complete [ 3895.267777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.272501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.277928] Lustre: Skipped 1 previous similar message [ 3899.576157] Lustre: server umount lustre-MDT0000 complete [ 3900.385355] LustreError: 128283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.406642] LustreError: 128283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3904.208112] LustreError: 128076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157081 with bad export cookie 13804640569995726352 [ 3904.218891] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.223503] LustreError: 128076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3905.008770] Lustre: server umount lustre-MDT0001 complete [ 3918.817971] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3922.594510] Key type lgssc unregistered [ 3922.985564] LNet: 130948:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3922.999456] LNetError: 130948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3923.030043] LNet: Removed LNI 192.168.204.120@tcp [ 3924.541741] Key type .llcrypt unregistered [ 3924.543376] Key type ._llcrypt unregistered [ 3954.634906] Key type ._llcrypt registered [ 3954.642294] Key type .llcrypt registered [ 3954.770834] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3972.139808] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3973.059415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3973.316158] alg: No test for adler32 (adler32-zlib) [ 3974.343156] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3974.612671] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3976.319344] Key type lgssc registered [ 3977.503689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4021.282138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4021.310133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.716699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4022.790149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4022.893070] Lustre: lustre-MDT0000: new disk, initializing [ 4022.977871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.004522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4027.669607] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4041.728265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.989875] Lustre: 134353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4042.125389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4042.134124] Lustre: Skipped 1 previous similar message [ 4042.313430] Lustre: lustre-MDT0001: new disk, initializing [ 4042.457994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4042.496191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4042.514911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4047.412198] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4058.948128] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.926222] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.516904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.830825] Lustre: lustre-OST0000: new disk, initializing [ 4075.834121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4075.838188] Lustre: 136322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4075.906986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.803133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4081.839276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.067276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4084.919681] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4098.128666] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.001463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.495426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4111.380669] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4111.777407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4117.987548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.995125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4118.001445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4118.004418] LustreError: Skipped 1 previous similar message [ 4118.012485] Lustre: Skipped 1 previous similar message [ 4121.375967] Lustre: server umount lustre-OST0000 complete [ 4128.228384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.251158] Lustre: Skipped 1 previous similar message [ 4128.253730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4128.266193] Lustre: Skipped 1 previous similar message [ 4132.135367] Lustre: server umount lustre-MDT0000 complete [ 4137.172885] LustreError: 134344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157314 with bad export cookie 16945245091703809861 [ 4137.187173] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4137.195483] LustreError: 134344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4137.908402] Lustre: server umount lustre-MDT0001 complete [ 4148.268542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.906491] LustreError: 138543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.924219] LustreError: 138543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4149.005167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.184051] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4154.338983] LustreError: 138544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.436907] LustreError: 138543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.552691] LustreError: 138544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.548957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.172265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.141640] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4177.667869] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4183.984467] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.446859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.743166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.964689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4206.659947] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.587835] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4211.862723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.666824] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4216.859774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4225.772175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.931488] Lustre: lustre-OST0001: new disk, initializing [ 4225.935154] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4225.941090] Lustre: 141914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4226.039203] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4232.109237] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4234.826889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4234.852438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4234.914524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4241.843795] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4242.146196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4247.507582] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4247.751829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4254.495549] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92bce0bfc000.ost_server_uuid 50 [ 4256.787239] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92bce0bfc000.ost_server_uuid in IDLE state after 0 sec [ 4271.075953] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.084056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4271.130752] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4271.132035] LustreError: Skipped 1 previous similar message [ 4271.158068] Lustre: Skipped 1 previous similar message [ 4272.974665] Lustre: server umount lustre-OST0001 complete [ 4276.205342] LustreError: 140363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.233736] LustreError: 140363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4280.805817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4280.825382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.844257] Lustre: Skipped 1 previous similar message [ 4280.863924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4286.438339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4286.440658] LustreError: 140363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.452137] Lustre: Skipped 3 previous similar messages [ 4286.465514] LustreError: 140363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4291.040197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4291.289922] Lustre: server umount lustre-OST0000 complete [ 4296.684426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.695329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4296.703262] Lustre: Skipped 2 previous similar messages [ 4296.728062] Lustre: Skipped 1 previous similar message [ 4301.632173] Lustre: server umount lustre-MDT0000 complete [ 4306.457063] LustreError: 138526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157483 with bad export cookie 16945245091703810743 [ 4306.461203] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4306.468496] LustreError: 138526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4306.971610] Lustre: server umount lustre-MDT0001 complete [ 4330.693614] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4334.327410] Key type lgssc unregistered [ 4334.733224] LNet: 144922:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4334.753027] LNetError: 144922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4334.773337] LNet: Removed LNI 192.168.204.120@tcp [ 4335.956650] Key type .llcrypt unregistered [ 4335.966951] Key type ._llcrypt unregistered [ 4377.915477] Key type ._llcrypt registered [ 4377.917883] Key type .llcrypt registered [ 4378.067712] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4394.866269] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4396.265576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4396.297300] alg: No test for adler32 (adler32-zlib) [ 4397.450593] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4397.790812] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4399.455654] Key type lgssc registered [ 4400.817412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4440.986084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4441.028540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.313762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4442.357545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4442.482029] Lustre: lustre-MDT0000: new disk, initializing [ 4442.585976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.616109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4448.162841] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4459.734109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.874979] Lustre: 149094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4459.940151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4459.943241] Lustre: Skipped 1 previous similar message [ 4460.070457] Lustre: lustre-MDT0001: new disk, initializing [ 4460.188584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4460.222227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4460.246433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4465.248751] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4476.408132] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4482.967157] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.351763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.652466] Lustre: lustre-OST0000: new disk, initializing [ 4490.659432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4490.676414] Lustre: 151061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4490.765823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.034539] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4499.551714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4499.567372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4499.697722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4508.739621] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4514.131145] Lustre: Failing over lustre-OST0000 [ 4514.153383] LustreError: 151867:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (00000000580f797c) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4514.362739] Lustre: server umount lustre-OST0000 complete [ 4531.681145] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157692/real 1788157692] req@0000000079e8360b x1875019116349056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788157708 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4531.727257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.220379] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157692/real 1788157692] req@000000001c8dea87 x1875019116349184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788157708 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4533.257857] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4533.688821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.072795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.100783] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4537.120528] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157697/real 1788157697] req@0000000064e75742 x1875019116350336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4541.014915] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4541.218749] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157702/real 1788157702] req@000000004866cc6f x1875019116351488/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4541.263640] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4541.267338] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4542.393753] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4542.399043] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4544.971973] Lustre: Failing over lustre-MDT0000 [ 4545.438716] Lustre: server umount lustre-MDT0000 complete [ 4546.412618] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.435901] Lustre: Skipped 3 previous similar messages [ 4547.425845] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157707/real 1788157707] req@00000000749cb5ac x1875019116352512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4547.461898] Lustre: 147319:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4551.652438] LustreError: 150316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.669183] LustreError: 150316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4552.480784] LustreError: 149099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.773268] LustreError: 149104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.810927] LustreError: 149104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4561.893061] LustreError: 149099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.911357] LustreError: 149099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4562.913985] Lustre: 147317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157723/real 1788157723] req@00000000c8d55ec7 x1875019116364800/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788157739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4562.943629] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.625631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.714114] LustreError: 149099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.753479] LustreError: 149099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4573.390557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4573.444568] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4573.868418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4578.675229] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4578.797265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4578.806419] Lustre: Skipped 1 previous similar message [ 4578.848805] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4578.917898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4588.516297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.522225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.532585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4589.034324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4589.041508] Lustre: Skipped 1 previous similar message [ 4594.153623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4594.542658] Lustre: server umount lustre-OST0000 complete [ 4599.266432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.269892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.272652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.272660] Lustre: Skipped 1 previous similar message [ 4599.281079] Lustre: Skipped 2 previous similar messages [ 4604.386928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4604.394173] Lustre: Skipped 1 previous similar message [ 4604.995664] Lustre: server umount lustre-MDT0000 complete [ 4609.505954] LustreError: 149104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.545421] LustreError: 149104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4609.962333] LustreError: 151060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157786 with bad export cookie 15470615981717535859 [ 4609.969077] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.987057] LustreError: 151060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4610.537047] Lustre: server umount lustre-MDT0001 complete [ 4624.307143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4627.883646] Key type lgssc unregistered [ 4628.271362] LNet: 154718:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4628.289970] LNetError: 154718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4628.325411] LNet: Removed LNI 192.168.204.120@tcp [ 4629.854522] Key type .llcrypt unregistered [ 4629.859258] Key type ._llcrypt unregistered [ 4671.018786] Key type ._llcrypt registered [ 4671.020755] Key type .llcrypt registered [ 4671.111499] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4672.551381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4672.592738] alg: No test for adler32 (adler32-zlib) [ 4673.713770] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4673.987390] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4675.727304] Key type lgssc registered [ 4676.839428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4684.716441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4684.734036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.210470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4690.257971] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4696.549591] LustreError: 156240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.572206] LustreError: 156240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4700.516604] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4701.671612] LustreError: 156241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.884180] LustreError: 156269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4704.894296] LustreError: 156269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4704.899649] LustreError: 156269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4707.220522] Lustre: server umount lustre-MDT0000 complete [ 4721.979340] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4735.236294] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4773.454518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.678772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4773.708287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4773.785961] Lustre: lustre-MDT0000: new disk, initializing [ 4773.875717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.907628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4777.747479] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4786.637424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.765894] Lustre: 159576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4786.841129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4786.844706] Lustre: Skipped 1 previous similar message [ 4786.931503] Lustre: lustre-MDT0001: new disk, initializing [ 4787.054283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4787.098371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4787.106448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4791.124269] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4800.506789] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.972805] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4814.224779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.663865] Lustre: lustre-OST0000: new disk, initializing [ 4814.675529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4814.703201] Lustre: 161546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4814.856868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.237969] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4824.692664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4824.715954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4824.872759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4834.085891] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4838.412932] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4841.049910] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:33:34 (1788158014) [ 5132.257135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.259063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5132.267307] Lustre: Skipped 2 previous similar messages [ 5132.281029] Lustre: Skipped 2 previous similar messages [ 5137.396863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5137.420068] Lustre: Skipped 2 previous similar messages [ 5142.495724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5142.503976] Lustre: Skipped 2 previous similar messages [ 5145.055527] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5145.262916] Lustre: server umount lustre-MDT0000 complete [ 5147.635488] LustreError: 159582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.675912] LustreError: 159582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5150.042470] LustreError: 160371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158326 with bad export cookie 13548502888972551045 [ 5150.048665] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5150.065868] LustreError: 160371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5152.736273] LustreError: 163330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.738407] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.783071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5156.384878] Lustre: server umount lustre-MDT0001 complete [ 5166.270196] Lustre: server umount lustre-OST0000 complete [ 5177.401801] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5188.873491] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5226.947666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.109633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5227.136875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5227.260411] Lustre: lustre-MDT0000: new disk, initializing [ 5227.327942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.345655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5231.662829] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5242.398084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.557848] Lustre: 166214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5242.620485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5242.632647] Lustre: Skipped 1 previous similar message [ 5242.746170] Lustre: lustre-MDT0001: new disk, initializing [ 5242.875623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.949778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5242.969430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5247.321247] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5256.772495] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5263.194307] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5270.364933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.745367] Lustre: lustre-OST0000: new disk, initializing [ 5270.753261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5270.761361] Lustre: 168183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5270.873719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.166795] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5279.326940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5279.340593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5279.428865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5286.717426] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5291.293880] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5291.522720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5296.365775] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5296.613334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5300.198623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.204450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5300.205827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5300.205833] Lustre: Skipped 1 previous similar message [ 5300.211250] Lustre: Skipped 1 previous similar message [ 5304.726987] Lustre: server umount lustre-OST0000 complete [ 5309.410966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5309.421401] LustreError: Skipped 1 previous similar message [ 5309.428971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.444986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5309.450957] Lustre: Skipped 1 previous similar message [ 5314.609987] Lustre: server umount lustre-MDT0000 complete [ 5315.552249] LustreError: 168642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.571057] LustreError: 168642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5318.783436] LustreError: 166208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158495 with bad export cookie 13548502888972709868 [ 5318.787530] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5318.801936] LustreError: 166208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5319.391449] Lustre: server umount lustre-MDT0001 complete [ 5329.483906] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:41:44 (1788158504) [ 5347.980335] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5358.496486] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5399.512621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.741206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5399.773995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5399.848806] Lustre: lustre-MDT0000: new disk, initializing [ 5399.931100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.949992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5403.857748] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5414.773128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.012695] Lustre: 172944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.547680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5420.804730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5433.822954] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5442.804780] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5449.889566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.153356] Lustre: lustre-OST0000: new disk, initializing [ 5450.156963] Lustre: Skipped 1 previous similar message [ 5450.160457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5450.164083] Lustre: Skipped 2 previous similar messages [ 5450.167464] Lustre: 174922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5456.139563] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5457.571257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5457.585238] Lustre: Skipped 1 previous similar message [ 5457.603924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5457.776429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5466.174316] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5473.247971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5473.249521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.253777] LustreError: Skipped 1 previous similar message [ 5473.264534] Lustre: Skipped 1 previous similar message [ 5473.278955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.287368] Lustre: Skipped 3 previous similar messages [ 5479.396830] Lustre: server umount lustre-OST0000 complete [ 5483.489778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.514292] Lustre: Skipped 1 previous similar message [ 5489.873469] Lustre: server umount lustre-MDT0000 complete [ 5493.728855] LustreError: 172955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.748102] LustreError: 172955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5493.800972] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158670 with bad export cookie 13548502888972710757 [ 5493.801829] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.812779] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5494.182534] Lustre: server umount lustre-MDT0001 complete [ 5502.942217] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:44:37 (1788158677) [ 5521.177852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5532.544648] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5569.016565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.195692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5569.221843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5569.315887] Lustre: lustre-MDT0000: new disk, initializing [ 5569.407588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.412287] Lustre: Skipped 2 previous similar messages [ 5569.427582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5574.003052] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5584.600710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.698706] Lustre: 179251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5584.773536] LustreError: 179261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.958884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5589.486136] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5598.940942] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5604.918404] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5611.762335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.006966] Lustre: 181227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5618.819354] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5619.335103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5619.497735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5630.122080] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.164175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5640.165055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.167082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5640.167091] Lustre: Skipped 6 previous similar messages [ 5640.213930] Lustre: Skipped 2 previous similar messages [ 5644.607613] Lustre: server umount lustre-OST0000 complete [ 5654.446585] Lustre: server umount lustre-MDT0000 complete [ 5655.526190] LustreError: 179257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.551216] LustreError: 179257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5658.407956] LustreError: 179242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158835 with bad export cookie 13548502888972711842 [ 5658.409614] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.420363] LustreError: 179242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5658.706151] Lustre: server umount lustre-MDT0001 complete [ 5670.441466] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5673.608850] Key type lgssc unregistered [ 5673.949061] LNet: 183083:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5673.972138] LNetError: 183083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5673.995316] LNet: Removed LNI 192.168.204.120@tcp [ 5674.906575] Key type .llcrypt unregistered [ 5674.912126] Key type ._llcrypt unregistered [ 5699.570840] Key type ._llcrypt registered [ 5699.573304] Key type .llcrypt registered [ 5699.684478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5700.913331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5701.093356] alg: No test for adler32 (adler32-zlib) [ 5702.182037] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5702.533378] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5704.264707] Key type lgssc registered [ 5705.557081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5713.248444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5713.270847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.734328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.705988] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5725.158321] LustreError: 184512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.175902] LustreError: 184512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5727.942186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.335814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5733.726865] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5741.977275] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5747.866766] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5754.865231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.223665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5760.319402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5761.565521] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5770.571285] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5794.276092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.289966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.302726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5799.348967] Lustre: server umount lustre-OST0000 complete [ 5804.518264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.531191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.535612] Lustre: Skipped 2 previous similar messages [ 5804.547367] Lustre: Skipped 2 previous similar messages [ 5809.634967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.640732] Lustre: Skipped 1 previous similar message [ 5809.859567] Lustre: server umount lustre-MDT0000 complete [ 5814.629989] LustreError: 186355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158991 with bad export cookie 1675829747917086640 [ 5814.634793] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.649368] LustreError: 186355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5814.989569] Lustre: server umount lustre-MDT0001 complete [ 5829.014518] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5831.985658] Key type lgssc unregistered [ 5832.325477] LNet: 188065:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5832.331585] LNetError: 188065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5832.352860] LNet: Removed LNI 192.168.204.120@tcp [ 5833.098745] Key type .llcrypt unregistered [ 5833.100733] Key type ._llcrypt unregistered [ 5862.137651] Key type ._llcrypt registered [ 5862.143835] Key type .llcrypt registered [ 5862.362256] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5864.401614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5864.430862] alg: No test for adler32 (adler32-zlib) [ 5865.457200] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5865.607917] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5867.263555] Key type lgssc registered [ 5868.840919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5876.528976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.542655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.944091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.693765] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5887.461965] LustreError: 189488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.485155] LustreError: 189488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5891.916068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.312290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.849235] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5904.409783] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5910.213089] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5917.305347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.892428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.036506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5925.781830] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5935.814607] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5943.609789] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 5943.779638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5943.796420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.804041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.809077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.831163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.847165] Lustre: Skipped 1 previous similar message [ 5946.862055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.871018] Lustre: Skipped 1 previous similar message [ 5949.377331] Lustre: server umount lustre-OST0000 complete [ 5958.373964] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92bcc63ad000.ost_server_uuid 50 [ 5960.438516] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5976.552666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.559030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.573399] Lustre: Skipped 1 previous similar message [ 5976.595371] Lustre: Skipped 2 previous similar messages [ 5981.666805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.682029] Lustre: Skipped 1 previous similar message [ 5981.780419] Lustre: server umount lustre-MDT0000 complete [ 5985.554085] LustreError: 189468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159162 with bad export cookie 14048822393990106280 [ 5985.565535] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5985.569123] LustreError: 189468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5986.009217] Lustre: server umount lustre-MDT0001 complete [ 5996.294533] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:52:50 (1788159170) [ 6004.054068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.785809] LustreError: 193405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.879809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.757246] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6010.337846] LustreError: 193406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.434210] LustreError: 193405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.923345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.333762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6024.281897] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6033.173892] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6039.498957] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6047.494645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6048.152544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.212527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6055.672954] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6063.858651] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6071.828446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.078863] Lustre: lustre-OST0001: new disk, initializing [ 6072.082613] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6072.089516] Lustre: 196249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6072.229911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.164871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6079.172632] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6079.306539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6079.693477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6090.298570] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6094.818535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.845718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.850959] Lustre: Skipped 1 previous similar message [ 6094.882619] Lustre: Skipped 1 previous similar message [ 6099.760910] Lustre: server umount lustre-OST0000 complete [ 6099.962059] LustreError: 195230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.976505] LustreError: 195230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6106.457854] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6106.621490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6110.189420] LustreError: 195229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.209455] LustreError: 195229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6110.689722] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6110.876091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6130.659074] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6130.659161] LustreError: 195230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.668730] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.676117] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6130.676126] Lustre: Skipped 1 previous similar message [ 6130.676241] LustreError: Skipped 1 previous similar message [ 6130.729875] LustreError: 195230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6132.135289] Lustre: server umount lustre-OST0001 complete [ 6137.314227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6140.896936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.910201] Lustre: Skipped 2 previous similar messages [ 6142.879660] Lustre: server umount lustre-MDT0000 complete [ 6146.588673] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159323 with bad export cookie 14048822393990107351 [ 6146.595546] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.599996] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6147.140935] Lustre: server umount lustre-MDT0001 complete [ 6169.909995] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:55:44 (1788159344) [ 6176.501600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.810119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6176.836881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6177.020816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.507189] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6190.651680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.768376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6190.802290] Lustre: 199690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6195.653726] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6204.506814] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.011416] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6218.804604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.012899] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6225.972804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6227.562370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6233.908070] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.506578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.640662] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6241.931370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.938151] Lustre: Skipped 2 previous similar messages [ 6248.169505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6248.313548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6257.706649] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.266117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.280456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.295369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.310215] Lustre: Skipped 4 previous similar messages [ 6267.410761] Lustre: server umount lustre-OST0000 complete [ 6268.710457] LustreError: 201540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.731324] LustreError: 201540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6290.661762] Lustre: server umount lustre-OST0001 complete [ 6300.819433] Lustre: server umount lustre-MDT0000 complete [ 6304.760199] LustreError: 199682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159481 with bad export cookie 14048822393990108758 [ 6304.762094] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.780144] LustreError: 199682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6305.339676] Lustre: server umount lustre-MDT0001 complete [ 6333.392617] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:58:27 (1788159507) [ 6349.723238] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6363.008546] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6406.188666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.423677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6406.462507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6406.577555] Lustre: lustre-MDT0000: new disk, initializing [ 6406.700701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.728121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6411.403778] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6422.226696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.349636] Lustre: 207789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6422.402604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6422.414404] Lustre: Skipped 1 previous similar message [ 6422.494893] Lustre: lustre-MDT0001: new disk, initializing [ 6422.671462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6422.687761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6427.265416] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6438.742911] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6445.197812] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6452.993982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.308457] Lustre: lustre-OST0000: new disk, initializing [ 6453.314401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6453.323088] Lustre: 209759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6460.403664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6460.506034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6460.519663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6460.766641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6471.024757] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.765769] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6476.999979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6482.142600] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6482.379183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6486.496559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.497689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.509993] LustreError: Skipped 2 previous similar messages [ 6486.520034] Lustre: Skipped 5 previous similar messages [ 6486.523580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6486.539400] Lustre: Skipped 9 previous similar messages [ 6490.581960] Lustre: server umount lustre-OST0000 complete [ 6501.179471] Lustre: server umount lustre-MDT0000 complete [ 6501.857477] LustreError: 207801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.875992] LustreError: 207801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6504.906397] LustreError: 207781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159681 with bad export cookie 14048822393990110543 [ 6504.912728] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6504.931116] LustreError: 207781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6505.249788] Lustre: server umount lustre-MDT0001 complete [ 6513.966949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.762104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6527.267425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.276916] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6538.814395] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6544.178826] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6550.900962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6557.468893] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6564.337249] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.187153] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6569.394561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6574.009559] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6574.209192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.145467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6578.152384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.159704] Lustre: Skipped 3 previous similar messages [ 6582.396270] Lustre: server umount lustre-OST0000 complete [ 6589.181885] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6593.907507] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6623.221915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6629.692644] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6630.707274] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.743113] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6637.163227] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.989328] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6645.540415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6645.547575] Lustre: Skipped 6 previous similar messages [ 6646.674227] Lustre: server umount lustre-OST0000 complete [ 6650.337188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6650.349966] LustreError: Skipped 1 previous similar message [ 6656.690901] Lustre: server umount lustre-MDT0000 complete [ 6660.677386] LustreError: 217079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159837 with bad export cookie 14048822393990111425 [ 6660.682869] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6660.701541] LustreError: 217079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6661.109982] Lustre: server umount lustre-MDT0001 complete [ 6670.648937] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:04:05 (1788159845) [ 6677.451055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.918804] LustreError: 217950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.933052] LustreError: 217950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6678.003177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.012072] Lustre: Skipped 6 previous similar messages [ 6682.825959] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6693.227956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.433477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6708.727110] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.757284] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6725.873337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.228765] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6743.452829] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.786216] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6749.074748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6754.884220] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6755.149928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6763.000143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.200330] Lustre: lustre-OST0001: new disk, initializing [ 6763.203810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6763.208911] Lustre: 221320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6769.276470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6769.293042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6769.447798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6770.581773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6783.692374] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6790.935154] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6791.199222] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6796.406811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6796.607666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6800.365195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.370479] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.377295] LustreError: Skipped 1 previous similar message [ 6800.409062] Lustre: Skipped 5 previous similar messages [ 6805.192942] Lustre: server umount lustre-OST0001 complete [ 6813.010338] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6813.318600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6819.623217] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6819.857945] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6850.724675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.177179] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6856.205182] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6856.235760] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6856.241716] Lustre: Skipped 1 previous similar message [ 6858.431921] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6866.842929] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6869.160919] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6877.678612] Lustre: server umount lustre-OST0001 complete [ 6901.086785] Lustre: server umount lustre-OST0000 complete [ 6905.316119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.326027] Lustre: Skipped 17 previous similar messages [ 6911.549249] Lustre: server umount lustre-MDT0000 complete [ 6915.745038] LustreError: 217930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160092 with bad export cookie 14048822393990112384 [ 6915.748221] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.756059] LustreError: 217930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6930.399479] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6930.799700] Lustre: server umount lustre-MDT0001 complete [ 6958.107339] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:08:52 (1788160132) [ 6965.851033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.339355] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6966.365253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6971.974516] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6981.644337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.729973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6981.759171] Lustre: 226580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6986.793218] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6995.966278] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7002.618855] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7010.244223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7010.418252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7017.709759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7024.812438] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7034.002415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.099720] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7041.196220] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7048.658514] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.102442] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7055.187459] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7055.398217] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7059.510568] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7059.731236] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7064.932917] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92bcd831b800.ost_server_uuid 50 [ 7066.427888] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92bcd831b800.ost_server_uuid in IDLE state after 0 sec [ 7067.454892] Lustre: Permanently deactivating lustre-OST0001 [ 7067.476104] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7070.757657] Lustre: Permanently reactivating lustre-OST0001 [ 7078.369681] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.381691] LustreError: Skipped 2 previous similar messages [ 7078.392673] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.414043] Lustre: Skipped 6 previous similar messages [ 7080.848360] Lustre: server umount lustre-OST0001 complete [ 7084.012231] LustreError: 228426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.023319] LustreError: 228426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 7091.006325] Lustre: server umount lustre-OST0000 complete [ 7109.087430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7109.439372] Lustre: server umount lustre-MDT0000 complete [ 7113.643299] LustreError: 226574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160290 with bad export cookie 14048822393990113672 [ 7113.644936] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.657700] LustreError: 226574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7114.067978] Lustre: server umount lustre-MDT0001 complete [ 7145.806192] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:12:00 (1788160320) [ 7154.273530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.620285] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7154.681983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7154.697338] Lustre: Skipped 5 previous similar messages [ 7160.031624] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7169.903050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.074592] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7170.152182] Lustre: 233274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7175.836826] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7184.974558] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.874789] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.938459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.217154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7200.240583] Lustre: Permanently deactivating lustre-OST0000 [ 7200.251961] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7200.435895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.447224] Lustre: Skipped 10 previous similar messages [ 7205.970368] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7206.076659] LustreError: 235434:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 7206.081820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7207.818349] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7217.052658] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7225.829450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.957816] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7232.306610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7234.143579] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7242.884817] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.061118] Lustre: Permanently reactivating lustre-OST0000 [ 7245.077105] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7245.081702] Lustre: Skipped 2 previous similar messages [ 7252.073244] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7252.079917] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7252.100547] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7252.170462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7252.185781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7270.879126] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7270.994114] Lustre: server umount lustre-OST0001 complete [ 7289.312165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7289.530215] Lustre: server umount lustre-OST0000 complete [ 7300.181686] Lustre: server umount lustre-MDT0000 complete [ 7303.198532] LustreError: 234015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160480 with bad export cookie 14048822393990115324 [ 7303.208436] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.214994] LustreError: 234015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7303.569885] Lustre: server umount lustre-MDT0001 complete [ 7315.420568] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7318.362974] Key type lgssc unregistered [ 7318.744682] LNet: 238138:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7318.751877] LNetError: 238138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7318.765172] LNet: Removed LNI 192.168.204.120@tcp [ 7319.604174] Key type .llcrypt unregistered [ 7319.606325] Key type ._llcrypt unregistered [ 7345.835081] Key type ._llcrypt registered [ 7345.843806] Key type .llcrypt registered [ 7345.991512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7347.673448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7347.704091] alg: No test for adler32 (adler32-zlib) [ 7348.826740] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7349.098430] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7350.799231] Key type lgssc registered [ 7352.217381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7362.835131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7362.849780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.351966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.854641] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7374.834683] LustreError: 239605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.861895] LustreError: 239605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7379.135606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.330510] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7379.375808] Lustre: 239595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7379.386945] Lustre: Permanently deactivating lustre-MDT0001 [ 7379.401470] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7379.695255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7384.412579] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7389.295927] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7394.641599] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7402.426979] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7410.884386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.359880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.708937] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7425.699758] LustreError: 241452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.617909] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7431.164045] LustreError: 242071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.253859] LustreError: 242070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.269462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7435.277733] LustreError: 242070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7435.648982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.877776] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7440.902225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7440.913099] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7442.863271] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7452.781322] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7456.043538] Lustre: Permanently reactivating lustre-MDT0001 [ 7456.068804] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7456.076942] Lustre: Skipped 2 previous similar messages [ 7462.215829] Lustre: lustre-MDT0001: Client c2ebcdcf-74ba-46c6-bcd7-a0876363849d (at 192.168.204.20@tcp) reconnecting [ 7463.147072] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.173243] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7463.186095] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7463.203879] LustreError: 243178:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7463.205416] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7463.222889] LustreError: 243178:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7463.223049] LustreError: 243178:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -108 [ 7463.224789] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7463.224789] [ 7463.287350] LustreError: 239598:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7469.209362] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7470.760429] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7477.383885] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7477.597636] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7481.936973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7482.163630] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7484.843979] Lustre: Permanently deactivating lustre-MDT0001 [ 7484.884221] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7484.889073] Lustre: Skipped 2 previous similar messages [ 7494.236177] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7504.361079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.362640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7504.368719] Lustre: Skipped 2 previous similar messages [ 7504.376429] Lustre: Skipped 3 previous similar messages [ 7509.472802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7509.477824] Lustre: Skipped 2 previous similar messages [ 7509.859907] Lustre: server umount lustre-MDT0000 complete [ 7513.411540] LustreError: 239587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160690 with bad export cookie 15973772144366645165 [ 7513.413449] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7513.434932] LustreError: 239587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7514.093968] Lustre: server umount lustre-MDT0001 complete [ 7520.471817] Lustre: server umount lustre-OST0000 complete [ 7535.325142] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:18:29 (1788160709) [ 7547.480098] Lustre: server umount lustre-OST0001 complete [ 7555.267484] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7568.104820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7607.988247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.437646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7608.446686] Lustre: Skipped 2 previous similar messages [ 7608.476537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7608.561894] Lustre: lustre-MDT0000: new disk, initializing [ 7608.706211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.735385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7613.612550] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7626.451560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.672664] Lustre: 247924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7626.766439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7626.771381] Lustre: Skipped 1 previous similar message [ 7626.930971] Lustre: lustre-MDT0001: new disk, initializing [ 7627.094314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7627.116406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7627.126462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7632.499898] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7642.835551] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7649.711695] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7657.638129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.957742] Lustre: lustre-OST0000: new disk, initializing [ 7657.960516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7657.965396] Lustre: 249892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7658.048538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.687192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7663.710778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7663.879065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7665.748651] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7676.409542] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.787698] LustreError: 247929:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7687.438318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.574459] Lustre: lustre-OST0001: new disk, initializing [ 7687.578310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7687.583444] Lustre: 251008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7688.875700] LustreError: 247929:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7688.906566] LustreError: 247930:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7693.495807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7693.505973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7693.661145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7694.832548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7698.960302] LustreError: 247930:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7705.279876] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7707.120482] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7709.153696] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7709.154749] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.175241] LustreError: Skipped 1 previous similar message [ 7709.177500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7709.232096] Lustre: Skipped 2 previous similar messages [ 7714.286206] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7714.290324] Lustre: Skipped 1 previous similar message [ 7715.244716] Lustre: server umount lustre-OST0001 complete [ 7719.400213] LustreError: 249877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.418167] LustreError: 249877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7729.643354] LustreError: 249875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.656949] LustreError: 249875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7734.762723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.781367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7734.786411] Lustre: Skipped 1 previous similar message [ 7739.114610] Lustre: server umount lustre-OST0000 complete [ 7744.237853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.245965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7744.253214] Lustre: Skipped 2 previous similar messages [ 7744.263889] Lustre: Skipped 1 previous similar message [ 7750.786384] Lustre: server umount lustre-MDT0000 complete [ 7754.727849] LustreError: 250827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.765244] LustreError: 250827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7755.435923] LustreError: 247916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160932 with bad export cookie 15973772144366647762 [ 7755.456961] LustreError: 247916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7755.465879] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.047370] Lustre: server umount lustre-MDT0001 complete [ 7774.235911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7777.034770] Key type lgssc unregistered [ 7777.366944] LNet: 253044:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7777.379796] LNetError: 253044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7777.403119] LNet: Removed LNI 192.168.204.120@tcp [ 7778.348764] Key type .llcrypt unregistered [ 7778.358177] Key type ._llcrypt unregistered [ 7822.435546] Key type ._llcrypt registered [ 7822.441156] Key type .llcrypt registered [ 7822.589511] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7823.894340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7823.933589] alg: No test for adler32 (adler32-zlib) [ 7825.057501] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7825.285883] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7826.928760] Key type lgssc registered [ 7828.327812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7836.634516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7836.665678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.022544] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7838.064232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7838.288488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7843.221124] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7854.146887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.319902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7854.373393] Lustre: 255251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7854.800026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.648661] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7867.774385] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7874.334750] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7881.849988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.169053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7882.343400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.982763] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7890.547661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7897.409128] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7951.843576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7951.861194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.899035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.389826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.394147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7952.434057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.445132] Lustre: Skipped 1 previous similar message [ 7954.932395] Lustre: server umount lustre-OST0000 complete [ 7960.579497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7973.587817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.959599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.031541] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7978.048569] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7980.045670] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7986.835077] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.359884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7992.373920] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8028.134912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8028.135106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.142875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8028.150656] Lustre: Skipped 1 previous similar message [ 8028.158013] Lustre: Skipped 1 previous similar message [ 8032.420760] Lustre: server umount lustre-OST0000 complete [ 8038.368808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.388392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8038.393773] Lustre: Skipped 1 previous similar message [ 8042.788662] Lustre: server umount lustre-MDT0000 complete [ 8046.926196] LustreError: 255244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161223 with bad export cookie 10422057366849321231 [ 8046.929858] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8046.930498] LustreError: 255244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8047.273496] Lustre: server umount lustre-MDT0001 complete [ 8058.643437] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8061.112748] Key type lgssc unregistered [ 8061.433757] LNet: 260678:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8061.451879] LNetError: 260678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8062.500944] LNet: Removed LNI 192.168.204.120@tcp [ 8063.315926] Key type .llcrypt unregistered [ 8063.320643] Key type ._llcrypt unregistered [ 8086.188679] Key type ._llcrypt registered [ 8086.191825] Key type .llcrypt registered [ 8086.314771] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8087.506326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8087.619565] alg: No test for adler32 (adler32-zlib) [ 8088.827988] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8089.082832] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8090.775188] Key type lgssc registered [ 8092.057649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8100.025714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8100.043550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.585191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.190973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8112.106459] LustreError: 262112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.140679] LustreError: 262112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8117.218407] LustreError: 262111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.583376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.092668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8122.408344] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8130.772262] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.426136] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8147.962228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8148.616965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.026479] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8167.053861] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8172.177462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8198.122601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.124531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8198.136558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8198.170540] Lustre: Skipped 1 previous similar message [ 8201.519701] Lustre: server umount lustre-OST0000 complete [ 8208.352886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.369106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.379741] Lustre: Skipped 1 previous similar message [ 8208.416503] Lustre: Skipped 2 previous similar messages [ 8213.092327] Lustre: server umount lustre-MDT0000 complete [ 8213.472852] LustreError: 262112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.495209] LustreError: 262112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8218.601126] LustreError: 262111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.626026] LustreError: 262111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8218.652122] LustreError: 262092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161395 with bad export cookie 13258330549924462531 [ 8218.653959] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8218.660309] LustreError: 262092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8219.184693] Lustre: server umount lustre-MDT0001 complete [ 8232.458077] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8235.587927] Key type lgssc unregistered [ 8235.956695] LNet: 266423:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8235.967888] LNetError: 266423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8235.983796] LNet: Removed LNI 192.168.204.120@tcp [ 8236.730116] Key type .llcrypt unregistered [ 8236.732984] Key type ._llcrypt unregistered [ 8254.271605] Key type ._llcrypt registered [ 8254.272846] Key type .llcrypt registered [ 8254.386787] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8255.445624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8255.478106] alg: No test for adler32 (adler32-zlib) [ 8257.090244] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8257.408227] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8259.199542] Key type lgssc registered [ 8260.579770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8269.413402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8269.443303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.030863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.524355] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8280.544640] LustreError: 267645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.582426] LustreError: 267645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8285.671362] LustreError: 267646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8286.639695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.980735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.793654] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8300.207412] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8306.903259] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8315.628996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8315.820695] LustreError: 269472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.223329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.527456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8323.481486] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8333.099469] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8347.107305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.109100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.115659] LustreError: Skipped 1 previous similar message [ 8347.119224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.127542] Lustre: Skipped 1 previous similar message [ 8347.167816] Lustre: Skipped 1 previous similar message [ 8352.229563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8352.247121] Lustre: Skipped 1 previous similar message [ 8352.975397] Lustre: server umount lustre-OST0000 complete [ 8358.884426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8358.898253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.906693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8362.464930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8362.470159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8362.529682] Lustre: Skipped 1 previous similar message [ 8367.584391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.593027] Lustre: Skipped 1 previous similar message [ 8372.191425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8372.483880] Lustre: server umount lustre-MDT0000 complete [ 8372.709047] LustreError: 268374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.739080] LustreError: 268374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8377.088583] LustreError: 267626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161553 with bad export cookie 6027563478454246274 [ 8377.098197] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.101781] LustreError: 267626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8377.600252] Lustre: server umount lustre-MDT0001 complete [ 8390.491266] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8393.691194] Key type lgssc unregistered [ 8394.051769] LNet: 271331:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8394.062609] LNetError: 271331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8394.077738] LNet: Removed LNI 192.168.204.120@tcp [ 8395.150732] Key type .llcrypt unregistered [ 8395.153426] Key type ._llcrypt unregistered [ 8420.721387] Key type ._llcrypt registered [ 8420.731440] Key type .llcrypt registered [ 8420.877510] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8421.668751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8421.704772] alg: No test for adler32 (adler32-zlib) [ 8422.845881] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8423.186141] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8424.873702] Key type lgssc registered [ 8426.072796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8435.360951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8435.398573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.130673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.551424] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8446.459069] LustreError: 272747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8446.495473] LustreError: 272747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8451.559289] LustreError: 272748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.686586] LustreError: 272747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.663502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.594103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.510880] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8473.212842] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8479.715206] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8487.250749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.577179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.066298] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8502.014849] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8507.076579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8532.969977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8532.993649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.009464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8533.017756] Lustre: Skipped 1 previous similar message [ 8537.041666] Lustre: server umount lustre-OST0000 complete [ 8543.204431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.210713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.217473] Lustre: Skipped 2 previous similar messages [ 8543.238798] Lustre: Skipped 1 previous similar message [ 8547.306185] Lustre: server umount lustre-MDT0000 complete [ 8548.326875] LustreError: 273847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.340346] LustreError: 273847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8551.125814] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161727 with bad export cookie 10578112305446345726 [ 8551.131907] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8551.134705] LustreError: 272729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8551.461724] Lustre: server umount lustre-MDT0001 complete [ 8563.122851] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8567.177409] Key type lgssc unregistered [ 8567.565911] LNet: 277059:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8567.583983] LNetError: 277059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8567.622908] LNet: Removed LNI 192.168.204.120@tcp [ 8568.528932] Key type .llcrypt unregistered [ 8568.530629] Key type ._llcrypt unregistered [ 8589.549826] Key type ._llcrypt registered [ 8589.560947] Key type .llcrypt registered [ 8589.769915] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8590.842647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8590.859623] alg: No test for adler32 (adler32-zlib) [ 8592.026922] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8592.384256] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8594.207180] Key type lgssc registered [ 8595.801525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8604.902720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8604.944540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.514736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.464587] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8616.937552] LustreError: 278300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.968795] LustreError: 278300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8621.271823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.790450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8626.300864] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8635.453729] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8642.888397] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8651.179224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8651.798693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.896162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8659.487682] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8667.848264] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8681.297589] Lustre: server umount lustre-OST0000 complete [ 8687.584931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.602215] Lustre: Skipped 1 previous similar message [ 8687.602956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8687.620843] Lustre: Skipped 1 previous similar message [ 8692.706076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8692.711210] Lustre: Skipped 1 previous similar message [ 8695.873619] Lustre: server umount lustre-MDT0000 complete [ 8697.829799] LustreError: 278313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.860979] LustreError: 278313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8698.850310] Lustre: 277595:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161859/real 1788161859] req@000000009fe1010a x1875023515119232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788161875 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8698.874788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.904750] LustreError: 278301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.922060] LustreError: 278301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8700.189426] LustreError: 278265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161877 with bad export cookie 17137223569551431113 [ 8700.211130] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.220391] LustreError: 278265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8700.995480] Lustre: server umount lustre-MDT0001 complete [ 8714.759228] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8718.324282] Key type lgssc unregistered [ 8718.620394] LNet: 282001:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8718.629477] LNetError: 282001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8718.645729] LNet: Removed LNI 192.168.204.120@tcp [ 8719.839275] Key type .llcrypt unregistered [ 8719.842464] Key type ._llcrypt unregistered [ 8733.252761] Key type ._llcrypt registered [ 8733.263569] Key type .llcrypt registered [ 8733.421163] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8753.543633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8769.243824] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8770.347550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8770.359539] alg: No test for adler32 (adler32-zlib) [ 8771.468728] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8771.787859] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8773.583182] Key type lgssc registered [ 8774.456223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8813.096966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8813.130939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.478436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8814.537951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8814.680714] Lustre: lustre-MDT0000: new disk, initializing [ 8814.831240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.874258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8820.711760] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8832.955901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.109465] Lustre: 285554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8833.230078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8833.239551] Lustre: Skipped 1 previous similar message [ 8833.466479] Lustre: lustre-MDT0001: new disk, initializing [ 8833.613465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8833.646203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8833.656638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8839.922728] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8851.875944] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8858.175345] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8866.621285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.865447] Lustre: lustre-OST0000: new disk, initializing [ 8866.871557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8866.880225] Lustre: 287525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8866.983413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.086438] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8874.561647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8874.573887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8874.757315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8885.299550] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8890.409445] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8890.658392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8895.303230] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8895.519504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8900.579909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.581388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8900.612244] LustreError: Skipped 1 previous similar message [ 8900.619131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8900.629498] Lustre: Skipped 1 previous similar message [ 8903.666133] Lustre: server umount lustre-OST0000 complete [ 8910.308570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8910.323130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.341581] Lustre: Skipped 1 previous similar message [ 8910.347301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.839230] Lustre: server umount lustre-MDT0000 complete [ 8917.405857] LustreError: 285546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162094 with bad export cookie 9168389563699351003 [ 8917.409052] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8917.418384] LustreError: 285546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8917.699098] Lustre: server umount lustre-MDT0001 complete [ 8925.764796] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:41:40 (1788162100) [ 8932.970357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.518417] LustreError: 289941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.543983] LustreError: 289941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8933.588694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.346274] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8938.977458] LustreError: 289942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.099770] LustreError: 289941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.232831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.681148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8952.264677] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8959.701913] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8965.985983] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8973.430132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.867150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.123865] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8986.860819] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9002.468225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.468548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.469861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.469867] Lustre: Skipped 1 previous similar message [ 9002.499400] Lustre: Skipped 2 previous similar messages [ 9006.613750] Lustre: server umount lustre-OST0000 complete [ 9012.704584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.726862] Lustre: Skipped 1 previous similar message [ 9012.727451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.741904] Lustre: Skipped 2 previous similar messages [ 9017.824964] LustreError: 290669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.843880] LustreError: 290669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9017.880364] Lustre: server umount lustre-MDT0000 complete [ 9021.916839] LustreError: 289922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162198 with bad export cookie 9168389563699351885 [ 9021.919374] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.938189] LustreError: 289922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9022.167621] Lustre: server umount lustre-MDT0001 complete [ 9035.010457] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9037.755918] Key type lgssc unregistered [ 9038.033590] LNet: 293501:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9038.042631] LNetError: 293501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9038.061821] LNet: Removed LNI 192.168.204.120@tcp [ 9038.615342] Key type .llcrypt unregistered [ 9038.617422] Key type ._llcrypt unregistered [ 9077.409757] Key type ._llcrypt registered [ 9077.414037] Key type .llcrypt registered [ 9077.614267] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9093.068744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9094.063818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9094.159324] alg: No test for adler32 (adler32-zlib) [ 9095.269512] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9095.524509] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9097.183212] Key type lgssc registered [ 9098.365468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9143.626673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9143.640747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.805414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9144.835920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9144.901286] Lustre: lustre-MDT0000: new disk, initializing [ 9145.000732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.053631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9148.889945] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9158.822530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.981618] Lustre: 297328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9159.034269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9159.041566] Lustre: Skipped 1 previous similar message [ 9159.129936] Lustre: lustre-MDT0001: new disk, initializing [ 9159.247480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9159.284165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9159.296929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9164.133131] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9175.567356] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9182.964960] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9190.606162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.131515] Lustre: lustre-OST2710: new disk, initializing [ 9191.135427] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9191.143367] Lustre: 299295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.272277] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9197.113911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9197.125765] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9197.275923] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9199.442664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9213.102626] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9220.195932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9220.399241] Lustre: lustre-OST03e8: new disk, initializing [ 9220.408401] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9220.417167] Lustre: 300363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9220.518311] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9227.270829] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9230.569629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9230.587503] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9230.671016] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9237.215978] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9245.055871] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9245.260653] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9249.833113] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9250.044334] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9254.577764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9254.809435] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.484519] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9259.817542] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9266.609648] Lustre: server umount lustre-MDT0000 complete [ 9266.672139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9266.673859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.708535] Lustre: Skipped 1 previous similar message [ 9271.115537] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162447 with bad export cookie 4872157126672726141 [ 9271.135205] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.137331] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9271.454324] Lustre: server umount lustre-MDT0001 complete [ 9285.953496] Lustre: server umount lustre-OST2710 complete [ 9287.649321] Lustre: 295125:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162448/real 1788162448] req@0000000055c4f136 x1875024042595200/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9287.682320] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.697820] Lustre: Skipped 2 previous similar messages [ 9290.375524] Lustre: server umount lustre-OST03e8 complete [ 9311.114841] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9321.618843] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9360.469110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.882087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9360.934161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9361.025065] Lustre: lustre-MDT0000: new disk, initializing [ 9361.091487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9361.108191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9366.743168] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9379.229279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.363225] Lustre: 305613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9379.436981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9379.444418] Lustre: Skipped 1 previous similar message [ 9379.575801] Lustre: lustre-MDT0001: new disk, initializing [ 9379.720128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9379.760557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9379.768979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9384.250939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9393.926694] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9400.773091] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9409.468770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9410.053611] Lustre: lustre-OST0000: new disk, initializing [ 9410.063209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9410.082371] Lustre: 307583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9410.267823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.984306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9418.337361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9418.349450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9418.550457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9427.427673] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9431.603453] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9431.791752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9435.408717] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9435.613244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9439.207621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9439.208722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.224816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9439.229989] Lustre: Skipped 1 previous similar message [ 9443.358297] Lustre: server umount lustre-OST0000 complete [ 9449.441801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.445538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9449.448547] Lustre: Skipped 2 previous similar messages [ 9449.468582] Lustre: Skipped 1 previous similar message [ 9452.723274] Lustre: server umount lustre-MDT0000 complete [ 9454.560425] LustreError: 305618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.575436] LustreError: 305618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9455.854803] LustreError: 307582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162632 with bad export cookie 4872157126672727821 [ 9455.863378] LustreError: 307582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9455.871122] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.131898] Lustre: server umount lustre-MDT0001 complete [ 9463.296701] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:50:38 (1788162638) [ 9464.870721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9466.609963] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:50:41 (1788162641) [ 9470.215671] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9495.320661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.501928] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9495.519690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9495.688939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9499.736264] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9508.556278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.646384] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9508.678375] Lustre: 311437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9514.284583] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9537.521912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.715814] Lustre: Denying initial registration attempt from nid 192.168.204.120@tcp, specified as failover [ 9537.722332] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9537.727432] LustreError: 312892:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9537.732701] LustreError: 312892:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9537.740478] LustreError: 312892:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9537.837627] Lustre: server umount lustre-OST0000 complete [ 9537.839568] LustreError: 312892:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9548.768444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.781940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.796195] Lustre: Skipped 1 previous similar message [ 9548.815664] Lustre: Skipped 1 previous similar message [ 9551.432606] Lustre: server umount lustre-MDT0000 complete [ 9553.892229] LustreError: 311445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.919584] LustreError: 311445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9555.708493] LustreError: 313262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162732 with bad export cookie 4872157126672728696 [ 9555.708493] LustreError: 311430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162732 with bad export cookie 4872157126672728696 [ 9555.708504] LustreError: 313262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9555.726936] LustreError: 311430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9555.729233] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9556.038847] Lustre: server umount lustre-MDT0001 complete [ 9567.159280] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9570.184985] Key type lgssc unregistered [ 9570.525334] LNet: 314014:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9570.545393] LNetError: 314014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9570.585739] LNet: Removed LNI 192.168.204.120@tcp [ 9571.693806] Key type .llcrypt unregistered [ 9571.697111] Key type ._llcrypt unregistered [ 9582.768589] Key type ._llcrypt registered [ 9582.771074] Key type .llcrypt registered [ 9582.904021] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9583.807697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9583.819119] alg: No test for adler32 (adler32-zlib) [ 9584.800202] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9584.956398] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9586.618477] Key type lgssc registered [ 9587.468849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9636.634533] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9644.860730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9644.877473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.073501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9646.092348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9646.251321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.448471] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9658.480882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.603327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9658.640635] Lustre: 316779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9658.836030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9662.593700] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9669.519630] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9674.547787] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9693.735626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.889169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9694.051314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9699.179723] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9707.511387] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9711.590431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9711.597809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.608591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9712.096679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9712.106233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9712.118471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9712.128880] Lustre: Skipped 1 previous similar message [ 9717.225965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.344730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.362873] Lustre: Skipped 1 previous similar message [ 9724.389477] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9724.777750] Lustre: server umount lustre-OST0000 complete [ 9730.531936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9730.542675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.579964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.584698] Lustre: Skipped 1 previous similar message [ 9732.589861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.608792] Lustre: server umount lustre-MDT0000 complete [ 9740.068983] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162916 with bad export cookie 14472181939075534592 [ 9740.079437] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.100524] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9740.828474] Lustre: server umount lustre-MDT0001 complete [ 9754.836127] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9757.957357] Key type lgssc unregistered [ 9758.330555] LNet: 320356:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9758.350599] LNetError: 320356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9759.402214] LNet: Removed LNI 192.168.204.120@tcp [ 9760.295653] Key type .llcrypt unregistered [ 9760.297694] Key type ._llcrypt unregistered [ 9784.533809] Key type ._llcrypt registered [ 9784.536955] Key type .llcrypt registered [ 9784.683083] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9785.177754] Key type .llcrypt unregistered [ 9785.181581] Key type ._llcrypt unregistered [ 9800.586706] Key type ._llcrypt registered [ 9800.590487] Key type .llcrypt registered [ 9800.684478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9801.600728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9801.659079] alg: No test for adler32 (adler32-zlib) [ 9802.673424] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9802.937554] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9804.615187] Key type lgssc registered [ 9805.834493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9839.383737] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [ 9840.882991] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [ 9853.918122] loop0: detected capacity change from 0 to 204800000 [ 9854.065598] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9854.076681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9854.085295] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9871.721201] loop0: detected capacity change from 0 to 204800000 [ 9871.870520] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9871.884962] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9871.896889] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9891.536596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9891.582945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.035881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9893.073875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9893.370389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.096254] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9908.217404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.372152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9908.410353] Lustre: 324283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9908.606453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9913.098762] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9921.962931] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9928.669367] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9934.409872] loop0: detected capacity change from 0 to 204800000 [ 9953.197555] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9953.530111] Lustre: sv7923-OST0000: new disk, initializing [ 9953.536397] Lustre: srv-sv7923-OST0000: No data found on store. Initialize space. [ 9953.547335] Lustre: 326362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9953.635527] Lustre: sv7923-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9960.457271] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9970.513578] loop1: detected capacity change from 0 to 204800000 [ 9991.278960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.491775] Lustre: sv7923-OST0001: new disk, initializing [ 9991.498731] Lustre: srv-sv7923-OST0001: No data found on store. Initialize space. [ 9991.504927] Lustre: 327481:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9991.589977] Lustre: sv7923-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.741417] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10013.458444] Lustre: server umount sv7923-OST0001 complete [10022.313174] Lustre: server umount sv7923-OST0000 complete [10038.756213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10038.791970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10038.809334] Lustre: Skipped 1 previous similar message [10043.888397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10044.652502] Lustre: server umount lustre-MDT0000 complete [10048.881228] LustreError: 327480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163225 with bad export cookie 2352150074234167443 [10048.882657] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10048.905993] LustreError: 327480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10049.421791] Lustre: server umount lustre-MDT0001 complete [10063.276447] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10066.101221] Key type lgssc unregistered [10066.416703] LNet: 330237:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10066.433127] LNetError: 330237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10066.448592] LNet: Removed LNI 192.168.204.120@tcp [10067.498323] LNet: Removed LNI 192.168.204.120@tcp1 [10068.308899] Key type .llcrypt unregistered [10068.312109] Key type ._llcrypt unregistered [10087.681797] Key type ._llcrypt registered [10087.684375] Key type .llcrypt registered [10087.828944] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10088.137977] Key type .llcrypt unregistered [10088.145354] Key type ._llcrypt unregistered [10105.000664] Key type ._llcrypt registered [10105.003202] Key type .llcrypt registered [10105.094772] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10106.137070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10106.221583] alg: No test for adler32 (adler32-zlib) [10107.316797] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10107.589585] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10109.263752] Key type lgssc registered [10110.233779] Lustre: Echo OBD driver; http://www.lustre.org/ [10133.157536] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [10134.386337] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [10144.708399] loop0: detected capacity change from 0 to 204800000 [10144.965580] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10144.993017] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10145.018644] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10162.928696] loop0: detected capacity change from 0 to 204800000 [10163.174251] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10163.199830] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10163.237408] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10182.032131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10182.071023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10183.468693] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10183.491928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10183.725915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10189.213658] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10198.365409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.463712] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10198.489927] Lustre: 334070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10198.786172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10203.433620] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10212.447921] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10219.355022] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10224.983446] loop0: detected capacity change from 0 to 204800000 [10243.630131] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.852477] Lustre: sv681-OST0000: new disk, initializing [10243.857058] Lustre: srv-sv681-OST0000: No data found on store. Initialize space. [10243.865818] Lustre: 336151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10243.929275] Lustre: sv681-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10250.588526] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10261.757636] loop1: detected capacity change from 0 to 204800000 [10281.466258] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10281.636777] Lustre: sv681-OST0001: new disk, initializing [10281.639498] Lustre: srv-sv681-OST0001: No data found on store. Initialize space. [10281.644054] Lustre: 337268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10281.676658] Lustre: sv681-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10288.821980] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10301.380172] Lustre: server umount sv681-OST0001 complete [10312.317846] Lustre: server umount sv681-OST0000 complete [10325.480068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.482385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10325.488257] Lustre: Skipped 1 previous similar message [10325.502932] Lustre: Skipped 1 previous similar message [10330.592592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10330.610045] Lustre: Skipped 1 previous similar message [10331.721121] Lustre: server umount lustre-MDT0000 complete [10335.899889] LustreError: 334812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163512 with bad export cookie 13258257310175389070 [10335.905715] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.912272] LustreError: 334812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10336.280423] Lustre: server umount lustre-MDT0001 complete [10351.179448] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10355.306762] Key type lgssc unregistered [10355.695682] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.20@tcp [10355.734379] LNet: 340022:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10355.751441] LNetError: 340022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10355.774979] LNet: Removed LNI 192.168.204.120@tcp [10356.842589] LNet: Removed LNI 192.168.204.120@tcp1 [10358.196385] Key type .llcrypt unregistered [10358.201375] Key type ._llcrypt unregistered [10379.207645] Key type ._llcrypt registered [10379.217423] Key type .llcrypt registered [10379.390809] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10379.705820] Key type .llcrypt unregistered [10379.710523] Key type ._llcrypt unregistered [10397.147176] Key type ._llcrypt registered [10397.152295] Key type .llcrypt registered [10397.273555] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10398.205527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10398.259979] alg: No test for adler32 (adler32-zlib) [10399.474392] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10399.860033] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10401.593092] Key type lgssc registered [10402.978626] Lustre: Echo OBD driver; http://www.lustre.org/ [10426.246808] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [10427.849341] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [10441.133588] loop0: detected capacity change from 0 to 204800000 [10441.440953] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10441.461108] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10441.483750] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10459.959757] loop0: detected capacity change from 0 to 204800000 [10460.207799] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10460.222287] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10460.242795] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10480.561779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10480.587264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10481.920448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10481.976987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10482.207098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10487.079469] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10496.528815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.704128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10496.762748] Lustre: 343861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10497.018983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10501.372966] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10510.009414] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10516.030229] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.509239] loop0: detected capacity change from 0 to 204800000 [10540.267542] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10540.525898] Lustre: sv9217-OST0000: new disk, initializing [10540.531338] Lustre: srv-sv9217-OST0000: No data found on store. Initialize space. [10540.546645] Lustre: 345940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10540.611442] Lustre: sv9217-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10547.664579] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10566.823715] Lustre: server umount sv9217-OST0000 complete [10584.049299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10584.057532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10584.091222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10585.059687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10585.060739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10587.546713] Lustre: server umount lustre-MDT0000 complete [10591.952391] LustreError: 343855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163768 with bad export cookie 18202435080624425028 [10591.970196] LustreError: 343855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10591.970257] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10592.605603] Lustre: server umount lustre-MDT0001 complete [10607.838357] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10610.398591] Key type lgssc unregistered [10610.666177] LNet: 348392:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10610.675984] LNetError: 348392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10610.681472] LNet: Removed LNI 192.168.204.120@tcp [10611.750080] LNet: Removed LNI 192.168.204.120@tcp1 [10612.676400] Key type .llcrypt unregistered [10612.685082] Key type ._llcrypt unregistered [10634.536697] Key type ._llcrypt registered [10634.539818] Key type .llcrypt registered [10634.680504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10634.947596] Key type .llcrypt unregistered [10634.948775] Key type ._llcrypt unregistered [10654.196871] Key type ._llcrypt registered [10654.199255] Key type .llcrypt registered [10654.290891] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10655.561819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10655.573101] alg: No test for adler32 (adler32-zlib) [10656.800353] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10657.004760] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10658.815189] Key type lgssc registered [10659.788488] Lustre: Echo OBD driver; http://www.lustre.org/ [10684.759266] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [10685.991137] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [10696.961322] loop0: detected capacity change from 0 to 204800000 [10697.182583] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10697.196821] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10697.219972] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10715.989295] loop0: detected capacity change from 0 to 204800000 [10716.317049] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10716.375768] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10716.399672] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10735.270633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.284058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.614943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10736.631133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10736.759479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10742.212796] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10752.869548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.034738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10753.062478] Lustre: 352267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10753.238145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10757.507756] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10764.913423] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10770.396807] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10775.835538] loop0: detected capacity change from 0 to 204800000 [10795.710800] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.076198] Lustre: sv17458-OST0000: new disk, initializing [10796.085773] Lustre: srv-sv17458-OST0000: No data found on store. Initialize space. [10796.111301] Lustre: 354342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10796.181238] Lustre: sv17458-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10802.310944] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10822.418894] Lustre: server umount sv17458-OST0000 complete [10840.545874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10840.556054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.565483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.058741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.090500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.097410] Lustre: Skipped 1 previous similar message [10843.056394] Lustre: server umount lustre-MDT0000 complete [10847.199115] LustreError: 352258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164024 with bad export cookie 16637650872951180454 [10847.205629] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10847.209833] LustreError: 352258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10847.649507] Lustre: server umount lustre-MDT0001 complete [10861.036598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10864.291780] Key type lgssc unregistered [10864.611880] LNet: 356795:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10864.624163] LNetError: 356795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10864.635884] LNet: Removed LNI 192.168.204.120@tcp [10865.706621] LNet: Removed LNI 192.168.204.120@tcp1 [10866.498177] Key type .llcrypt unregistered [10866.503461] Key type ._llcrypt unregistered [10885.916168] Key type ._llcrypt registered [10885.918745] Key type .llcrypt registered [10886.065467] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10886.454653] Key type .llcrypt unregistered [10886.461796] Key type ._llcrypt unregistered [10904.691864] Key type ._llcrypt registered [10904.693854] Key type .llcrypt registered [10904.838892] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10906.245372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10906.404369] alg: No test for adler32 (adler32-zlib) [10907.562862] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10907.907509] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10909.631216] Key type lgssc registered [10910.816780] Lustre: Echo OBD driver; http://www.lustre.org/ [10934.868735] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [10936.264214] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [10946.562328] loop0: detected capacity change from 0 to 204800000 [10946.709856] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10946.729811] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10946.741357] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10964.118839] loop0: detected capacity change from 0 to 204800000 [10964.245502] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10964.257138] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10964.267077] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10982.744279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10982.781638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.095148] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10984.119180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10984.261481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.112666] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10998.309080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.429975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10998.460591] Lustre: 360625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10998.751519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11004.000294] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11011.441413] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11017.550936] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11023.346546] loop0: detected capacity change from 0 to 204800000 [11041.962490] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11042.233334] Lustre: sv28279-OST0000: new disk, initializing [11042.238453] Lustre: srv-sv28279-OST0000: No data found on store. Initialize space. [11042.247588] Lustre: 362705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11042.305581] Lustre: sv28279-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.274481] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11058.777910] loop1: detected capacity change from 0 to 204800000 [11077.923265] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11078.112566] Lustre: sv28279-OST0001: new disk, initializing [11078.117646] Lustre: srv-sv28279-OST0001: No data found on store. Initialize space. [11078.123917] Lustre: 363872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11078.183276] Lustre: sv28279-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11084.620964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11098.446114] Lustre: server umount sv28279-OST0001 complete [11108.421950] Lustre: server umount sv28279-OST0000 complete [11121.635346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11121.644487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.661492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11123.681923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.684825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11123.703074] Lustre: Skipped 1 previous similar message [11126.719729] Lustre: server umount lustre-MDT0000 complete [11129.973311] LustreError: 364718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164306 with bad export cookie 3879431191795687138 [11129.978845] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11129.982748] LustreError: 364718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11130.366865] Lustre: server umount lustre-MDT0001 complete [11141.462189] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11143.701107] Key type lgssc unregistered [11143.928633] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.20@tcp [11143.956219] LNet: 366625:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11143.968282] LNetError: 366625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11143.976280] LNet: Removed LNI 192.168.204.120@tcp [11145.000315] LNet: Removed LNI 192.168.204.120@tcp1 [11145.799933] Key type .llcrypt unregistered [11145.801659] Key type ._llcrypt unregistered [11159.708080] Key type ._llcrypt registered [11159.710402] Key type .llcrypt registered [11159.804982] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11159.996549] Key type .llcrypt unregistered [11160.001303] Key type ._llcrypt unregistered [11177.425556] Key type ._llcrypt registered [11177.430872] Key type .llcrypt registered [11177.559831] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11178.761135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11178.883164] alg: No test for adler32 (adler32-zlib) [11180.195220] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11180.544222] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11182.223809] Key type lgssc registered [11183.517280] Lustre: Echo OBD driver; http://www.lustre.org/ [11203.557827] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [11204.962485] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [11237.715563] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11241.165919] Key type lgssc unregistered [11241.485493] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.20@tcp [11241.519090] LNet: 370587:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11241.524375] LNetError: 370587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11241.531916] LNet: Removed LNI 192.168.204.120@tcp [11241.538578] LNet: Removed LNI 192.168.204.120@tcp1 [11242.331065] Key type .llcrypt unregistered [11242.334904] Key type ._llcrypt unregistered [11262.391049] Key type ._llcrypt registered [11262.394058] Key type .llcrypt registered [11262.544230] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11263.166819] Key type .llcrypt unregistered [11263.173385] Key type ._llcrypt unregistered [11279.277598] Key type ._llcrypt registered [11279.280666] Key type .llcrypt registered [11279.395172] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11280.526804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11280.652027] alg: No test for adler32 (adler32-zlib) [11281.586936] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11281.867064] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11283.528846] Key type lgssc registered [11285.378643] Lustre: Echo OBD driver; http://www.lustre.org/ [11312.290279] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [11313.629745] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [11315.250328] LNet: Added LNI 192.168.204.120@tcp2 [8/256/0/180] [11327.059744] loop0: detected capacity change from 0 to 204800000 [11327.304912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11327.342958] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11327.365670] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11344.769182] loop0: detected capacity change from 0 to 204800000 [11345.009731] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11345.016372] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11345.033309] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11363.444929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11363.468964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.621084] LNetError: 371914:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.120@tcp NID 192.168.204.120@tcp is already locked with peer 192.168.204.120@tcp [11364.739763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11364.759191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11364.962467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11369.802748] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11379.762887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.843251] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11379.893650] Lustre: 374506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11380.066395] LNetError: 371914:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.120@tcp NID 192.168.204.120@tcp is already locked with peer 192.168.204.120@tcp [11380.078798] LNetError: 371914:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 1 previous similar message [11380.205498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11384.825362] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11388.435705] LNetError: 371914:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.120@tcp NID 192.168.204.120@tcp is already locked with peer 192.168.204.120@tcp [11388.449725] LNetError: 371914:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 2 previous similar messages [11393.288916] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11399.458191] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11405.047159] loop0: detected capacity change from 0 to 204800000 [11424.074330] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11424.648873] Lustre: sv13787-OST0000: new disk, initializing [11424.661320] Lustre: srv-sv13787-OST0000: No data found on store. Initialize space. [11424.682597] Lustre: 376583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11424.763282] Lustre: sv13787-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11431.296335] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11441.831805] loop1: detected capacity change from 0 to 204800000 [11459.362845] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11459.484184] Lustre: sv13787-OST0001: new disk, initializing [11459.488853] Lustre: srv-sv13787-OST0001: No data found on store. Initialize space. [11459.495694] Lustre: 377748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11459.546196] Lustre: sv13787-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11467.269698] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11483.165652] Lustre: server umount sv13787-OST0001 complete [11493.873347] Lustre: server umount sv13787-OST0000 complete [11508.207035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11508.224525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.235021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.264526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.268036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.964992] Lustre: server umount lustre-MDT0000 complete [11517.909671] LustreError: 374496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164694 with bad export cookie 13416522494058250053 [11517.911106] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11517.920566] LustreError: 374496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11518.307504] Lustre: server umount lustre-MDT0001 complete [11530.484848] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11533.005314] Key type lgssc unregistered [11533.311461] LNet: 380506:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11533.336646] LNetError: 380506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11533.344966] LNet: Removed LNI 192.168.204.120@tcp [11534.367264] LNet: Removed LNI 192.168.204.120@tcp1 [11534.384417] LNetError: 380506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11534.394783] LNetError: 380506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11534.425979] LNet: Removed LNI 192.168.204.120@tcp2 [11535.390984] Key type .llcrypt unregistered [11535.393153] Key type ._llcrypt unregistered [11556.069438] Key type ._llcrypt registered [11556.076302] Key type .llcrypt registered [11556.196345] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11556.459547] Key type .llcrypt unregistered [11556.465072] Key type ._llcrypt unregistered [11574.361433] Key type ._llcrypt registered [11574.363385] Key type .llcrypt registered [11574.451795] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11575.451976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.584110] alg: No test for adler32 (adler32-zlib) [11576.624661] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11576.939349] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11578.615174] Key type lgssc registered [11579.796702] Lustre: Echo OBD driver; http://www.lustre.org/ [11600.827879] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [11602.249653] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [11613.391843] loop0: detected capacity change from 0 to 204800000 [11613.551833] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11613.564451] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11613.583407] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11630.980328] loop0: detected capacity change from 0 to 204800000 [11631.165865] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11631.185600] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11631.211356] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11650.728512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11650.747373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11652.077221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11652.112078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11652.278444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11657.260689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11667.169306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.311364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11667.374195] Lustre: 384398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11667.693188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11672.488893] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11679.883238] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11686.628902] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11691.919928] loop0: detected capacity change from 0 to 204800000 [11711.338949] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11711.571654] Lustre: sv12608-OST0000: new disk, initializing [11711.575326] Lustre: srv-sv12608-OST0000: No data found on store. Initialize space. [11711.581083] Lustre: 386479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11711.634644] Lustre: sv12608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11718.883559] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11729.814440] loop1: detected capacity change from 0 to 204800000 [11748.788251] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11748.923745] Lustre: sv12608-OST0001: new disk, initializing [11748.927860] Lustre: srv-sv12608-OST0001: No data found on store. Initialize space. [11748.934449] Lustre: 387596:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11749.004281] Lustre: sv12608-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11754.311351] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11767.945935] Lustre: server umount sv12608-OST0001 complete [11777.531781] Lustre: server umount sv12608-OST0000 complete [11795.938047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11795.955828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.978359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.692472] Lustre: server umount lustre-MDT0000 complete [11802.519364] LustreError: 385141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164979 with bad export cookie 6316060449346202015 [11802.523768] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11802.528011] LustreError: 385141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11802.765815] Lustre: server umount lustre-MDT0001 complete [11816.144838] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11819.048578] Key type lgssc unregistered [11819.327043] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.20@tcp [11819.350749] LNet: 390349:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11819.356868] LNetError: 390349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11819.363071] LNet: Removed LNI 192.168.204.120@tcp [11820.388987] LNet: Removed LNI 192.168.204.120@tcp1 [11821.268801] Key type .llcrypt unregistered [11821.271285] Key type ._llcrypt unregistered [11841.660354] Key type ._llcrypt registered [11841.663379] Key type .llcrypt registered [11841.839296] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11842.261348] Key type .llcrypt unregistered [11842.265130] Key type ._llcrypt unregistered [11849.265978] Key type ._llcrypt registered [11849.268555] Key type .llcrypt registered [11849.360332] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11849.627304] Key type .llcrypt unregistered [11849.631372] Key type ._llcrypt unregistered [11861.754198] Key type ._llcrypt registered [11861.756764] Key type .llcrypt registered [11861.830688] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11862.039793] Key type .llcrypt unregistered [11862.043825] Key type ._llcrypt unregistered [11876.965901] Key type ._llcrypt registered [11876.968357] Key type .llcrypt registered [11877.039174] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11877.301411] Key type .llcrypt unregistered [11877.306696] Key type ._llcrypt unregistered [11890.246688] Key type ._llcrypt registered [11890.249856] Key type .llcrypt registered [11890.355161] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11891.363710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11891.418045] alg: No test for adler32 (adler32-zlib) [11892.459401] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11892.679577] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11894.335180] Key type lgssc registered [11895.378474] Lustre: Echo OBD driver; http://www.lustre.org/ [11915.735587] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [11916.890674] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [11927.614946] loop0: detected capacity change from 0 to 204800000 [11927.763717] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11927.777420] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11927.807790] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11945.718786] loop0: detected capacity change from 0 to 204800000 [11945.981798] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11945.995192] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.016406] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11964.842889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11964.860741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.020221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11966.033662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11966.165611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11970.412766] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11980.782060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.963151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11981.034636] Lustre: 395788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11981.377642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11987.251268] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11996.608988] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12003.423995] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12009.196363] loop0: detected capacity change from 0 to 204800000 [12027.973804] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12028.541342] Lustre: sv25586-OST0000: new disk, initializing [12028.569563] Lustre: srv-sv25586-OST0000: No data found on store. Initialize space. [12028.593607] Lustre: 397864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12028.727758] Lustre: sv25586-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12038.535516] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12051.200300] loop1: detected capacity change from 0 to 204800000 [12070.193427] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12070.339751] Lustre: sv25586-OST0001: new disk, initializing [12070.346677] Lustre: srv-sv25586-OST0001: No data found on store. Initialize space. [12070.359067] Lustre: 398985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12070.432052] Lustre: sv25586-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12078.535350] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12099.671253] Lustre: 399976:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12099.683041] Lustre: 399976:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12099.695683] Lustre: 399976:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12099.710652] Lustre: 399976:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12099.728094] Lustre: 399976:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12099.749836] Lustre: 399976:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12102.191247] Lustre: server umount sv25586-OST0001 complete [12111.902786] Lustre: server umount sv25586-OST0000 complete [12124.645234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12124.660930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.681187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.712747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.713612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.983049] Lustre: server umount lustre-MDT0000 complete [12134.478822] LustreError: 396529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165311 with bad export cookie 12682021764422584678 [12134.484592] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.488225] LustreError: 396529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12134.788583] Lustre: server umount lustre-MDT0001 complete [12149.781766] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12152.522942] Key type lgssc unregistered [12152.863070] LNet: 401981:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12152.882550] LNetError: 401981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12152.904092] LNet: Removed LNI 192.168.204.120@tcp [12153.967501] LNet: Removed LNI 192.168.204.120@tcp1 [12155.888380] Key type .llcrypt unregistered [12155.893621] Key type ._llcrypt unregistered [12175.599171] Key type ._llcrypt registered [12175.601250] Key type .llcrypt registered [12175.695575] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12175.931342] Key type .llcrypt unregistered [12175.934128] Key type ._llcrypt unregistered [12191.950515] Key type ._llcrypt registered [12191.952965] Key type .llcrypt registered [12192.045391] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12193.492269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12193.529464] alg: No test for adler32 (adler32-zlib) [12194.549705] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12194.770835] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12196.415921] Key type lgssc registered [12197.560571] Lustre: Echo OBD driver; http://www.lustre.org/ [12219.351174] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [12220.469915] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [12229.881235] loop0: detected capacity change from 0 to 204800000 [12230.032905] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12230.041402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12230.052406] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12247.094700] loop0: detected capacity change from 0 to 204800000 [12247.233812] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12247.243843] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12247.254729] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12265.050919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12265.108700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12266.483585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12266.499572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12266.690941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12272.350488] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12282.119813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.221263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12282.286152] Lustre: 405805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12282.546170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12286.269367] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12294.694049] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12301.223894] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12306.959397] loop0: detected capacity change from 0 to 204800000 [12326.461783] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12326.676868] Lustre: sv23451-OST0000: new disk, initializing [12326.680608] Lustre: srv-sv23451-OST0000: No data found on store. Initialize space. [12326.685260] Lustre: 407883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12326.750277] Lustre: sv23451-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12332.712243] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12341.528347] loop1: detected capacity change from 0 to 204800000 [12359.924639] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12360.112738] Lustre: sv23451-OST0001: new disk, initializing [12360.127179] Lustre: srv-sv23451-OST0001: No data found on store. Initialize space. [12360.133878] Lustre: 409002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12360.196611] Lustre: sv23451-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12367.588205] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12384.629887] Lustre: 409993:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12384.645330] Lustre: 409993:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12384.656526] Lustre: 409993:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12384.666843] Lustre: 409993:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12384.674429] Lustre: 409993:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12384.684103] Lustre: 409993:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12386.707162] Lustre: server umount sv23451-OST0001 complete [12396.755487] Lustre: server umount sv23451-OST0000 complete [12410.850313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12410.853047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.859407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12410.875438] Lustre: Skipped 1 previous similar message [12415.644103] Lustre: server umount lustre-MDT0000 complete [12419.643898] LustreError: 405798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165596 with bad export cookie 14707017278032815295 [12419.645584] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.655664] LustreError: 405798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12420.035185] Lustre: server umount lustre-MDT0001 complete [12432.230440] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12434.852530] Key type lgssc unregistered [12435.125519] LNet: 411998:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12435.130775] LNetError: 411998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12436.191333] LNet: Removed LNI 192.168.204.120@tcp [12436.195641] LNetError: 411998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12436.209094] LNet: Removed LNI 192.168.204.120@tcp1 [12436.917139] Key type .llcrypt unregistered [12436.919530] Key type ._llcrypt unregistered [12455.136859] Key type ._llcrypt registered [12455.141278] Key type .llcrypt registered [12455.233319] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12455.492541] Key type .llcrypt unregistered [12455.497780] Key type ._llcrypt unregistered [12470.652286] Key type ._llcrypt registered [12470.658332] Key type .llcrypt registered [12470.781143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12471.714693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12471.796350] alg: No test for adler32 (adler32-zlib) [12472.880388] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12473.168464] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12474.863197] Key type lgssc registered [12475.781101] Lustre: Echo OBD driver; http://www.lustre.org/ [12496.133666] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [12497.637388] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [12508.667182] loop0: detected capacity change from 0 to 204800000 [12508.828823] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12508.840797] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12508.850713] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12526.678918] loop0: detected capacity change from 0 to 204800000 [12526.796326] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12526.823258] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12526.839206] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12545.139671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12545.158920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12546.421278] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12546.436462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12546.607133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12550.897804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12558.991925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12559.096568] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12559.128246] Lustre: 415820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12559.334199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12563.636134] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12571.095929] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12578.205099] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12584.207548] loop0: detected capacity change from 0 to 204800000 [12602.920436] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12603.229657] Lustre: sv3602-OST0000: new disk, initializing [12603.235936] Lustre: srv-sv3602-OST0000: No data found on store. Initialize space. [12603.247191] Lustre: 417900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12603.325305] Lustre: sv3602-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12609.652508] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12620.012700] loop1: detected capacity change from 0 to 204800000 [12638.706023] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12638.854576] Lustre: sv3602-OST0001: new disk, initializing [12638.857554] Lustre: srv-sv3602-OST0001: No data found on store. Initialize space. [12638.866799] Lustre: 419020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12638.965284] Lustre: sv3602-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12646.088573] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12667.006818] Lustre: 420107:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12667.026167] Lustre: 420107:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12667.036115] Lustre: 420107:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12667.047805] Lustre: 420107:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12667.058199] Lustre: 420107:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12667.065745] Lustre: 420107:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12668.174680] Lustre: 420155:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12668.180688] Lustre: 420155:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [12668.185726] Lustre: 420155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12668.191734] Lustre: 420155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [12668.197214] Lustre: 420155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12668.202690] Lustre: 420155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [12668.208232] Lustre: 420155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12668.217687] Lustre: 420155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [12668.226503] Lustre: 420155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12668.233308] Lustre: 420155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [12668.238807] Lustre: 420155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12668.244484] Lustre: 420155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [12670.737636] Lustre: server umount sv3602-OST0001 complete [12680.430415] Lustre: server umount sv3602-OST0000 complete [12696.055183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.065411] Lustre: Skipped 1 previous similar message [12696.067238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.088371] Lustre: Skipped 1 previous similar message [12701.152875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12701.165950] Lustre: Skipped 1 previous similar message [12706.273227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12706.278452] Lustre: Skipped 1 previous similar message [12708.319156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12708.608770] Lustre: server umount lustre-MDT0000 complete [12712.361248] LustreError: 415813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165889 with bad export cookie 8404402173341912615 [12712.368591] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12712.378864] LustreError: 415813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12712.693969] Lustre: server umount lustre-MDT0001 complete [12723.428198] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12725.840514] Key type lgssc unregistered [12726.117358] LNet: 422162:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12726.133419] LNetError: 422162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12726.148190] LNet: Removed LNI 192.168.204.120@tcp [12727.208777] LNet: Removed LNI 192.168.204.120@tcp1 [12728.084238] Key type .llcrypt unregistered [12728.086567] Key type ._llcrypt unregistered [12751.449660] Key type ._llcrypt registered [12751.451969] Key type .llcrypt registered [12751.589485] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12751.873688] Key type .llcrypt unregistered [12751.877245] Key type ._llcrypt unregistered [12769.988857] Key type ._llcrypt registered [12769.992558] Key type .llcrypt registered [12770.167873] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12771.398912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12771.418863] alg: No test for adler32 (adler32-zlib) [12772.468317] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12772.679663] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12774.327510] Key type lgssc registered [12775.329721] Lustre: Echo OBD driver; http://www.lustre.org/ [12797.650837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [12799.240296] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [12810.857140] loop0: detected capacity change from 0 to 204800000 [12811.063630] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12811.090895] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12811.105816] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12829.218869] loop0: detected capacity change from 0 to 204800000 [12829.526558] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12829.547247] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12829.584594] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12848.927980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12848.961509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12850.321888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12850.372376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12850.582179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12855.198353] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12864.634229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12864.783906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12864.845537] Lustre: 425989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12865.192687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12869.429718] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12878.568832] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12886.256159] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12892.167984] loop0: detected capacity change from 0 to 204800000 [12913.007501] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12913.288736] Lustre: sv17376-OST0000: new disk, initializing [12913.294345] Lustre: srv-sv17376-OST0000: No data found on store. Initialize space. [12913.305308] Lustre: 428065:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12913.355709] Lustre: sv17376-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12919.674688] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12939.577659] Lustre: 429008:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12939.592474] Lustre: 429008:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12939.600056] Lustre: 429008:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12939.615718] Lustre: 429008:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12939.633620] Lustre: 429008:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12939.651390] Lustre: 429008:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12945.364456] Lustre: server umount sv17376-OST0000 complete [12959.200369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12959.210534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.323192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.330676] Lustre: Skipped 2 previous similar messages [12964.644381] Lustre: server umount lustre-MDT0000 complete [12968.882740] LustreError: 426738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166145 with bad export cookie 9159127353642250951 [12968.884321] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12968.896236] LustreError: 426738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12969.284796] Lustre: server umount lustre-MDT0001 complete [12983.528426] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12986.719412] Key type lgssc unregistered [12986.925805] LNet: 430716:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12986.930115] LNetError: 430716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12986.934387] LNet: Removed LNI 192.168.204.120@tcp [12988.010521] LNet: Removed LNI 192.168.204.120@tcp1 [12988.636210] Key type .llcrypt unregistered [12988.643865] Key type ._llcrypt unregistered [13006.208624] Key type ._llcrypt registered [13006.210732] Key type .llcrypt registered [13006.362973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13006.698774] Key type .llcrypt unregistered [13006.704118] Key type ._llcrypt unregistered [13024.510804] Key type ._llcrypt registered [13024.516539] Key type .llcrypt registered [13024.651716] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13026.446593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13026.497705] alg: No test for adler32 (adler32-zlib) [13027.504201] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13027.766846] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13029.447286] Key type lgssc registered [13030.432393] Lustre: Echo OBD driver; http://www.lustre.org/ [13051.446276] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [13053.038273] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [13055.308901] LNet: Added LNI 192.168.204.120@tcp2 [8/256/0/180] [13070.898362] loop0: detected capacity change from 0 to 204800000 [13071.176070] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13071.198806] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13071.216109] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13089.346068] loop0: detected capacity change from 0 to 204800000 [13089.645032] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13089.662773] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13089.682492] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13108.117096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13108.129822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13109.457339] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13109.507428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13109.753371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13114.426749] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13123.302242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13123.635962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13123.677872] Lustre: 434638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13124.131906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13128.504860] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13139.643031] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13147.159354] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13153.335373] loop0: detected capacity change from 0 to 204800000 [13173.352356] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13173.589066] Lustre: sv25115-OST0000: new disk, initializing [13173.601325] Lustre: srv-sv25115-OST0000: No data found on store. Initialize space. [13173.614714] Lustre: 436712:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13173.730357] Lustre: sv25115-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13180.517629] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13195.113675] loop1: detected capacity change from 0 to 204800000 [13214.437469] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13214.672887] Lustre: sv25115-OST0001: new disk, initializing [13214.675231] Lustre: srv-sv25115-OST0001: No data found on store. Initialize space. [13214.680848] Lustre: 437833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13214.766446] Lustre: sv25115-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13222.434180] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13233.197208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [13234.369547] Lustre: 438582:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13234.377109] Lustre: 438582:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13234.389214] Lustre: 438582:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13234.401859] Lustre: 438582:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13234.416451] Lustre: 438582:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13234.425329] Lustre: 438582:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13236.812969] Lustre: server umount sv25115-OST0001 complete [13247.039872] Lustre: server umount sv25115-OST0000 complete [13262.312376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13262.323663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.347416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13264.865375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13264.867692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13264.902674] Lustre: Skipped 1 previous similar message [13266.782637] Lustre: server umount lustre-MDT0000 complete [13270.718812] LustreError: 434629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166447 with bad export cookie 9368993791680922698 [13270.728190] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13270.731894] LustreError: 434629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13271.330953] Lustre: server umount lustre-MDT0001 complete [13283.721976] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13287.428368] Key type lgssc unregistered [13287.782548] LNet: 440589:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13287.796628] LNetError: 440589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13287.809562] LNet: Removed LNI 192.168.204.120@tcp [13288.863213] LNet: Removed LNI 192.168.204.120@tcp1 [13288.868217] LNetError: 440589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13288.880122] LNetError: 440589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13288.904602] LNet: Removed LNI 192.168.204.120@tcp2 [13290.132400] Key type .llcrypt unregistered [13290.135597] Key type ._llcrypt unregistered [13309.932558] Key type ._llcrypt registered [13309.936430] Key type .llcrypt registered [13310.006242] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13310.222525] Key type .llcrypt unregistered [13310.225722] Key type ._llcrypt unregistered [13327.341762] Key type ._llcrypt registered [13327.344349] Key type .llcrypt registered [13327.475572] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13328.449506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13328.560720] alg: No test for adler32 (adler32-zlib) [13329.784503] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13330.096867] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13331.783388] Key type lgssc registered [13333.037738] Lustre: Echo OBD driver; http://www.lustre.org/ [13355.626714] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [13357.050584] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [13368.265523] loop0: detected capacity change from 0 to 204800000 [13368.555572] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13368.566586] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13368.583627] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13386.103059] loop0: detected capacity change from 0 to 204800000 [13386.320796] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13386.330542] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13386.354076] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13404.430045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13404.449595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13405.679631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13405.704347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13405.833471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13410.189818] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13419.786543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13419.908307] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13419.974857] Lustre: 444464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13420.250078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13425.261672] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13433.759161] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13440.377661] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13446.175341] loop0: detected capacity change from 0 to 204800000 [13465.028079] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13465.464997] Lustre: sv5113-OST0000: new disk, initializing [13465.473939] Lustre: srv-sv5113-OST0000: No data found on store. Initialize space. [13465.488200] Lustre: 446540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13465.564981] Lustre: sv5113-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13473.793356] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13487.604977] loop1: detected capacity change from 0 to 204800000 [13508.152081] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13508.382770] Lustre: sv5113-OST0001: new disk, initializing [13508.386358] Lustre: srv-sv5113-OST0001: No data found on store. Initialize space. [13508.396179] Lustre: 447660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13508.495399] Lustre: sv5113-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13515.669963] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13527.202929] LustreError: 448552:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [13542.295130] LustreError: 448552:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [13551.638655] Lustre: 448846:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13551.648889] Lustre: 448846:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13551.663825] Lustre: 448846:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13551.670167] Lustre: 448846:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13551.676075] Lustre: 448846:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13551.683777] Lustre: 448846:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13553.653273] Lustre: server umount sv5113-OST0001 complete [13563.891488] Lustre: server umount sv5113-OST0000 complete [13579.232457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13579.248112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13579.268663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13580.769450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13580.772820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13580.797832] Lustre: Skipped 1 previous similar message [13583.562828] Lustre: server umount lustre-MDT0000 complete [13588.027586] LustreError: 444456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166764 with bad export cookie 13795910346517045759 [13588.034887] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13588.044206] LustreError: 444456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13588.509268] Lustre: server umount lustre-MDT0001 complete [13602.366562] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13605.570439] Key type lgssc unregistered [13605.881952] LNet: 450854:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13605.898384] LNetError: 450854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.908743] LNet: Removed LNI 192.168.204.120@tcp [13606.950585] LNet: Removed LNI 192.168.204.120@tcp1 [13607.960443] Key type .llcrypt unregistered [13607.963720] Key type ._llcrypt unregistered [13629.546706] Key type ._llcrypt registered [13629.553528] Key type .llcrypt registered [13629.703522] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13630.000363] Key type .llcrypt unregistered [13630.004214] Key type ._llcrypt unregistered [13649.844216] Key type ._llcrypt registered [13649.848548] Key type .llcrypt registered [13649.966477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13651.172240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13651.307076] alg: No test for adler32 (adler32-zlib) [13652.614705] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13652.925258] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13654.647216] Key type lgssc registered [13656.093562] Lustre: Echo OBD driver; http://www.lustre.org/ [13679.084349] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [13680.639960] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [13693.845863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13693.856685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13695.184321] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13695.234415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13695.479275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13700.320195] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13711.654081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13711.782636] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13711.817174] Lustre: 454138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13711.986886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13716.942643] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13725.589930] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13732.872995] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13737.601121] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [13738.025915] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.204.120@tcp1 [13739.658544] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/test_logs/1788166913 [13747.680597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13747.694753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13747.727387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.729736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.733342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13753.060105] Lustre: server umount lustre-MDT0000 complete [13757.451669] LustreError: 454133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166934 with bad export cookie 6586431573552539127 [13757.453755] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13757.475551] LustreError: 454133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13757.852304] Lustre: server umount lustre-MDT0001 complete [13774.733534] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13777.610551] Key type lgssc unregistered [13777.991859] LNet: 457485:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13777.999847] LNetError: 457485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13778.014388] LNet: Removed LNI 192.168.204.120@tcp [13779.055264] LNet: Removed LNI 192.168.204.120@tcp1 [13780.121922] Key type .llcrypt unregistered [13780.131493] Key type ._llcrypt unregistered [13799.513462] Key type ._llcrypt registered [13799.515058] Key type .llcrypt registered [13799.658112] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13800.937186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13801.181316] alg: No test for adler32 (adler32-zlib) [13802.327288] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13802.546101] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13804.223867] Key type lgssc registered [13805.339149] Lustre: Echo OBD driver; http://www.lustre.org/ [13811.640440] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [13812.731904] LNet: Added LNI 192.168.204.120@tcp1 [8/256/0/180] [13819.291207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13819.318443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13820.811344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13825.230303] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13831.136727] LustreError: 458965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13831.142067] LustreError: 458965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13833.187039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13833.747287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13840.612253] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13851.565319] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13859.958787] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13867.935147] LNet: Removed LNI 192.168.204.120@tcp1 [13868.868620] Lustre: 460676:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13868.878137] Lustre: 460676:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13868.888745] Lustre: 460676:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13868.898803] Lustre: 460676:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13868.911672] Lustre: 460676:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13868.922432] Lustre: 460676:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13874.657272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13874.657273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13874.659771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13874.705630] Lustre: Skipped 1 previous similar message [13879.780334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13879.791204] Lustre: Skipped 1 previous similar message [13880.667933] Lustre: server umount lustre-MDT0000 complete [13884.820840] LustreError: 458945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167061 with bad export cookie 11551619424817237926 [13884.830831] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13885.194626] Lustre: server umount lustre-MDT0001 complete [13895.896408] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13898.644831] Key type lgssc unregistered [13899.006420] LNet: 461682:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13899.025542] LNetError: 461682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13899.044688] LNet: Removed LNI 192.168.204.120@tcp [13899.819697] Key type .llcrypt unregistered [13899.823096] Key type ._llcrypt unregistered [13920.067875] Key type ._llcrypt registered [13920.070641] Key type .llcrypt registered [13920.177459] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13921.271150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13921.416219] alg: No test for adler32 (adler32-zlib) [13922.527105] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13922.852541] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13924.535207] Key type lgssc registered [13925.513837] Lustre: Echo OBD driver; http://www.lustre.org/ [13932.571626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13932.598037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13934.082658] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [13934.128975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13938.247236] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13944.295857] LustreError: 463000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13944.314968] LustreError: 463000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13946.390328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13946.547543] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [13946.558834] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13946.647069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13951.091520] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13958.171785] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13963.572235] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13972.336248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13972.625508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13972.888613] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [13972.905474] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13972.948508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13979.489571] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13980.184670] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [13980.194120] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13986.635642] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13996.001754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13996.008710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13996.016834] Lustre: Skipped 1 previous similar message [13996.023454] Lustre: Skipped 2 previous similar messages [13998.905780] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [14001.121270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14001.134290] Lustre: Skipped 2 previous similar messages [14001.528661] Lustre: server umount lustre-MDT0000 complete [14004.014058] LustreError: 462994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14005.060147] LustreError: 462981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167181 with bad export cookie 10593491390563680739 [14005.071647] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14005.073149] LustreError: 462981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14006.240081] LustreError: 465506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14006.244969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14006.272644] Lustre: Skipped 1 previous similar message [14006.276579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14009.130635] LustreError: 465506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14011.472562] Lustre: server umount lustre-MDT0001 complete [14018.475748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14028.999962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14029.797782] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9303a44b01b51fb1 [14029.821970] Lustre: MGC192.168.204.120@tcp: Connection restored to 0@lo (at 0@lo) [14030.210837] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14030.233353] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14030.264568] LustreError: 464828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14030.317592] LustreError: 464828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14030.418486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14030.552818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [14036.277922] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14039.657325] LustreError: 464829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14039.661084] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14039.668609] LustreError: 464829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14045.189766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14045.494576] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14045.508391] LNetError: 462309:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14045.618836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14050.509547] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14050.810870] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14059.537342] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14061.806913] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14068.275187] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14069.980367] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14076.384115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14076.401624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14076.413616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14076.434627] Lustre: Skipped 1 previous similar message [14076.461625] Lustre: Skipped 3 previous similar messages [14086.626534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14086.636831] Lustre: Skipped 3 previous similar messages [14090.211896] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14090.346501] Lustre: server umount lustre-OST0000 complete [14096.868595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14096.870103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14096.891467] LustreError: Skipped 1 previous similar message [14100.752806] Lustre: server umount lustre-MDT0000 complete [14101.984812] LustreError: 466610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14102.022351] LustreError: 466610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14105.582048] LustreError: 462980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167282 with bad export cookie 10593491390563688369 [14105.588510] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14105.600888] LustreError: 462980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14106.189941] Lustre: server umount lustre-MDT0001 complete [14120.816265] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14123.989711] Key type lgssc unregistered [14124.271238] LNet: 469283:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14124.279714] LNetError: 469283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14124.294793] LNet: Removed LNI 192.168.204.120@tcp [14125.236090] Key type .llcrypt unregistered [14125.238869] Key type ._llcrypt unregistered [14152.983458] Key type ._llcrypt registered [14152.985315] Key type .llcrypt registered [14153.133367] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14154.150916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14154.177207] alg: No test for adler32 (adler32-zlib) [14155.326364] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14155.646055] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [14157.311278] Key type lgssc registered [14158.454852] Lustre: Echo OBD driver; http://www.lustre.org/ [14165.702323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14165.734952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14167.121810] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14167.203479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14172.378523] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14176.741635] LustreError: 470702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14176.772859] LustreError: 470702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14181.865226] LustreError: 470701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14182.002381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14182.170049] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14182.179081] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14182.280753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14187.226214] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14194.864543] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14201.824869] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14207.974153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14207.983495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14207.989339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14208.036058] Lustre: Skipped 1 previous similar message [14213.089797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14213.095454] Lustre: Skipped 2 previous similar messages [14218.208182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14219.231814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14219.521494] Lustre: server umount lustre-MDT0000 complete [14223.332844] LustreError: 472222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14223.366203] LustreError: 472222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14223.380760] LustreError: 470683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167400 with bad export cookie 7526001490461698283 [14223.384326] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14223.401973] LustreError: 470683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14223.699513] Lustre: server umount lustre-MDT0001 complete [14237.808985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14238.070863] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14238.083669] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14238.185798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14238.252310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14238.481174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14243.497400] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14252.321461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14252.445355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14252.492945] Lustre: 473289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14252.640875] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14252.646458] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14252.744791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14256.952399] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14264.370646] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14270.141138] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14280.576862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14280.814487] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [14280.826207] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [14280.839913] LustreError: 475173:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [14280.843689] LustreError: 475173:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [14280.847654] LustreError: 475173:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [14280.923781] Lustre: server umount lustre-OST0000 complete [14280.929373] LustreError: 475173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [14291.120314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14291.332661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14291.526592] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14291.535596] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14291.570310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14297.672630] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14301.685526] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14301.698319] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14301.837309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [14305.650627] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14314.142046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14314.264645] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [14314.344634] Lustre: lustre-OST0001: new disk, initializing [14314.349870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14314.360107] Lustre: 476717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14314.472817] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14320.783452] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14324.287892] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.120@tcp1 has no available nets [14324.294948] LNetError: 470010:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [14324.332275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14324.344069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14324.493346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14328.834415] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14334.958802] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14334.968692] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14334.970055] Lustre: Skipped 1 previous similar message [14334.970480] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14334.976719] Lustre: Skipped 1 previous similar message [14334.996112] LustreError: Skipped 1 previous similar message [14338.446625] Lustre: server umount lustre-OST0001 complete [14340.081123] LustreError: 475641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14340.110911] LustreError: 475641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14345.192129] LustreError: 475642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14345.196037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14345.199750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14345.199758] Lustre: Skipped 1 previous similar message [14345.222633] LustreError: 475642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14345.243751] Lustre: Skipped 1 previous similar message [14349.654982] Lustre: server umount lustre-OST0000 complete [14355.425315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14355.436678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14355.442559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14355.442567] Lustre: Skipped 1 previous similar message [14355.443308] Lustre: Skipped 1 previous similar message [14359.999556] Lustre: server umount lustre-MDT0000 complete [14360.546383] LustreError: 477195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14360.573724] LustreError: 477195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14363.632690] LustreError: 475669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167540 with bad export cookie 7526001490461698836 [14363.634787] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14363.645714] LustreError: 475669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14364.002362] Lustre: server umount lustre-MDT0001 complete [14375.193817] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14377.863381] Key type lgssc unregistered [14378.185716] LNet: 478675:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14378.192579] LNetError: 478675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14378.213685] LNet: Removed LNI 192.168.204.120@tcp [14379.281441] Key type .llcrypt unregistered [14379.285523] Key type ._llcrypt unregistered [14455.233526] Key type ._llcrypt registered [14455.240567] Key type .llcrypt registered [14455.371340] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [14470.803939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14472.191269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14472.224393] alg: No test for adler32 (adler32-zlib) [14473.291594] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14473.598987] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [14475.288052] Key type lgssc registered [14476.794355] Lustre: Echo OBD driver; http://www.lustre.org/ [14518.418976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14518.452317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14519.724958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14519.772210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14519.854447] Lustre: lustre-MDT0000: new disk, initializing [14519.920394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14519.934985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14524.434691] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14536.449476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14536.633768] Lustre: 483614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14536.718527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14536.726614] Lustre: Skipped 1 previous similar message [14536.841286] Lustre: lustre-MDT0001: new disk, initializing [14536.953715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14537.004499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14537.014040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14541.295308] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14550.381486] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14556.223553] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14563.026485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14563.414220] Lustre: lustre-OST0000: new disk, initializing [14563.424828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14563.432756] Lustre: 485580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14563.548645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14569.939633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14572.651460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14572.665188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14572.757501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14579.881571] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14585.168828] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14585.455836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14590.635546] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14590.880695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14593.509614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14593.510489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14593.525203] LustreError: Skipped 1 previous similar message [14593.531862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14593.559425] Lustre: Skipped 1 previous similar message [14593.577687] Lustre: Skipped 1 previous similar message [14598.636131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14598.642838] Lustre: Skipped 1 previous similar message [14603.745117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14603.754089] Lustre: Skipped 1 previous similar message [14607.327159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14607.499628] Lustre: server umount lustre-OST0000 complete [14613.984969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14613.999147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14614.015602] Lustre: Skipped 1 previous similar message [14614.034493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14617.547702] Lustre: server umount lustre-MDT0000 complete [14621.758386] LustreError: 484417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167798 with bad export cookie 14510408085090872261 [14621.762816] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14621.765783] LustreError: 484417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14622.194787] Lustre: server umount lustre-MDT0001 complete [14631.534726] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:16:46 (1788167806) [14639.580934] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:16:54 (1788167814) [14648.377947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14648.906450] LustreError: 488338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14648.937375] LustreError: 488338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14648.996561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14654.361781] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14654.436350] LustreError: 488339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14658.534328] LustreError: 488338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14663.605145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14663.648645] LustreError: 488339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14663.934903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14668.659854] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14677.934286] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14684.963194] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14691.041602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14691.419095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14698.551832] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14706.391911] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14718.190393] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [14720.178755] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [14722.061267] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [14724.281508] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [14726.278280] Lustre: Failing over lustre-MDT0000 [14726.671954] Lustre: server umount lustre-MDT0000 complete [14729.009478] LustreError: 489482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14729.698561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14729.717968] Lustre: Skipped 2 previous similar messages [14739.240394] LustreError: 490833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14739.256138] LustreError: 490833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14746.079200] Lustre: 481834:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167906/real 1788167906] req@00000000a0058fe1 x1875029681840000/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788167922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14746.117681] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14746.209098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14755.317521] LustreError: 488339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14755.334044] LustreError: 488339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [14755.579307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14755.632408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14760.308731] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14760.751523] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14760.942056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14760.970940] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [14761.029106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [14769.934975] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14771.792598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14775.220540] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [14776.825037] Lustre: Failing over lustre-MDT0000 [14777.041942] Lustre: server umount lustre-MDT0000 complete [14781.411665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14781.415833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14781.427456] Lustre: Skipped 1 previous similar message [14791.462899] LustreError: 488333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14791.482558] LustreError: 488333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [14795.050335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14795.293779] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14795.594102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14795.659402] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14796.576038] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14799.665383] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14800.876318] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14800.889447] Lustre: Skipped 2 previous similar messages [14800.927652] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [14800.978234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [14809.119966] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14810.853801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14821.350987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14821.357445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14821.364154] Lustre: Skipped 2 previous similar messages [14821.367729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14821.396708] Lustre: Skipped 3 previous similar messages [14823.061741] Lustre: server umount lustre-MDT0000 complete [14827.196393] LustreError: 488319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168004 with bad export cookie 14510408085090874970 [14827.203953] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14827.204658] LustreError: 488319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14827.530624] Lustre: server umount lustre-MDT0001 complete [14841.327362] Lustre: server umount lustre-OST0000 complete [14857.753106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14858.065904] LustreError: 494704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14858.088702] LustreError: 494704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [14863.379564] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14872.913179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14873.309066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14873.314579] Lustre: Skipped 1 previous similar message [14877.127134] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14884.755066] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14891.192898] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14899.354572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.768789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [14906.728657] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14916.604921] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14928.361166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14928.371413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14928.391458] Lustre: Skipped 1 previous similar message [14928.395857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14933.629854] Lustre: server umount lustre-OST0000 complete [14944.840955] Lustre: server umount lustre-MDT0000 complete [14949.081750] LustreError: 494685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168125 with bad export cookie 14510408085090875719 [14949.088611] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14949.099748] LustreError: 494685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14949.340513] Lustre: server umount lustre-MDT0001 complete [14961.126429] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14964.309166] Key type lgssc unregistered [14964.683747] LNet: 498238:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14964.702737] LNetError: 498238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14964.733314] LNet: Removed LNI 192.168.204.120@tcp [14965.898886] Key type .llcrypt unregistered [14965.901164] Key type ._llcrypt unregistered [14993.089371] Key type ._llcrypt registered [14993.091774] Key type .llcrypt registered [14993.187201] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14994.344046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14994.378053] alg: No test for adler32 (adler32-zlib) [14995.523590] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14995.737046] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [14997.447352] Key type lgssc registered [14998.749956] Lustre: Echo OBD driver; http://www.lustre.org/ [15006.559549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15006.596448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15008.086753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15012.818696] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15017.448204] LustreError: 499762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15017.465990] LustreError: 499762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15022.057162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15022.454556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15028.287282] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15038.067505] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15044.437926] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15051.546156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15051.716041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15052.782735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [15057.672646] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15066.111068] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15080.539557] Lustre: server umount lustre-MDT0000 complete [15084.000353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15084.020827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15084.076427] LustreError: 499763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15084.469341] LustreError: 499742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168261 with bad export cookie 5089507462113915254 [15084.471186] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15084.484947] LustreError: 499742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15084.513095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15084.518377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15084.541946] Lustre: Skipped 1 previous similar message [15084.919669] Lustre: server umount lustre-MDT0001 complete [15091.339345] Lustre: server umount lustre-OST0000 complete [15106.072948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15106.449085] LustreError: 503494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15106.479414] LustreError: 503494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15106.557625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15111.467190] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15111.650965] LustreError: 503495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15116.773179] LustreError: 503494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15121.081263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15121.604068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15126.389198] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15135.027735] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15142.317547] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15149.176901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15149.543022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15151.633104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [15156.812875] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15165.610659] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15177.718062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15177.729372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15177.747171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15182.312397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15182.316973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15182.330555] Lustre: Skipped 1 previous similar message [15182.339906] Lustre: Skipped 1 previous similar message [15184.044887] Lustre: server umount lustre-MDT0000 complete [15187.437068] LustreError: 503494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15187.458562] LustreError: 503494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15188.295729] LustreError: 503475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168365 with bad export cookie 5089507462113916367 [15188.297804] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15188.325577] LustreError: 503475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15188.853582] Lustre: server umount lustre-MDT0001 complete [15204.042121] Lustre: server umount lustre-OST0000 complete [15225.729556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15226.386566] LustreError: 507404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15226.519905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15232.274972] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15242.205810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15249.815927] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15259.598982] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15266.961392] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15274.994781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15275.302326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15275.314559] Lustre: Skipped 1 previous similar message [15282.033271] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15291.242471] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15296.821194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [15302.113987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15302.119780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15302.152285] Lustre: Skipped 1 previous similar message [15302.166563] Lustre: Skipped 1 previous similar message [15304.203738] Lustre: server umount lustre-OST0000 complete [15309.282040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15309.304805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15309.322564] Lustre: Skipped 1 previous similar message [15312.355201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15312.362647] Lustre: Skipped 1 previous similar message [15314.768324] Lustre: server umount lustre-MDT0000 complete [15317.476375] LustreError: 508789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15317.515729] LustreError: 508789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15319.624510] LustreError: 507386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168496 with bad export cookie 5089507462113917389 [15319.632784] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15319.650953] LustreError: 507386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15320.233483] Lustre: server umount lustre-MDT0001 complete [15333.088993] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15336.651790] Key type lgssc unregistered [15337.061190] LNet: 510939:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15337.082362] LNetError: 510939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15337.106233] LNet: Removed LNI 192.168.204.120@tcp [15338.469287] Key type .llcrypt unregistered [15338.471023] Key type ._llcrypt unregistered [15369.410385] Key type ._llcrypt registered [15369.430467] Key type .llcrypt registered [15369.575388] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15371.451602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15371.541656] alg: No test for adler32 (adler32-zlib) [15372.741796] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15373.018383] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15374.767233] Key type lgssc registered [15376.286696] Lustre: Echo OBD driver; http://www.lustre.org/ [15385.715862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15385.733716] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15385.745172] LustreError: 512381:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [15385.763413] LustreError: 512381:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [15385.768346] LustreError: 512381:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [15385.773624] LustreError: 512381:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [15385.785705] LustreError: 512381:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15395.534911] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [15395.552236] LustreError: 512779:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [15395.570463] LustreError: 512779:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [15395.576985] LustreError: 512779:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [15395.585905] LustreError: 512779:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [15395.591594] LustreError: 512779:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15422.707116] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15425.828819] Key type lgssc unregistered [15426.051739] LNet: 513784:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15426.065224] LNetError: 513784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15426.091238] LNet: Removed LNI 192.168.204.120@tcp [15426.986561] Key type .llcrypt unregistered [15426.990182] Key type ._llcrypt unregistered [15449.587125] Key type ._llcrypt registered [15449.590398] Key type .llcrypt registered [15449.777749] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [15466.701605] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15467.660055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15467.691415] alg: No test for adler32 (adler32-zlib) [15468.902471] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15469.190623] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15470.919191] Key type lgssc registered [15472.044458] Lustre: Echo OBD driver; http://www.lustre.org/ [15513.730833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15513.805471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15515.083671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15515.132133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15515.240976] Lustre: lustre-MDT0000: new disk, initializing [15515.365435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15515.382460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15520.860667] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15533.366231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15533.547325] Lustre: 516941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15533.619697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15533.625361] Lustre: Skipped 1 previous similar message [15533.796954] Lustre: lustre-MDT0001: new disk, initializing [15533.947861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15534.004842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15534.020924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15539.502692] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15550.315545] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15556.901752] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15564.478923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15564.888977] Lustre: lustre-OST0000: new disk, initializing [15564.893624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15564.909893] Lustre: 518910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15565.037225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15572.099985] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15573.057410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15573.077058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15573.199366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15582.777811] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15587.401865] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15587.617867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15592.558323] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15592.857311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15599.077535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15599.078537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15599.109524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15599.125952] Lustre: Skipped 1 previous similar message [15600.961606] Lustre: server umount lustre-OST0000 complete [15605.731977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15605.743877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15605.760105] Lustre: Skipped 1 previous similar message [15605.768314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15609.312982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15609.323281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15610.931064] Lustre: server umount lustre-MDT0000 complete [15614.862831] LustreError: 516934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168791 with bad export cookie 8196802928302915706 [15614.865320] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15614.872531] LustreError: 516934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15615.238348] Lustre: server umount lustre-MDT0001 complete [15623.335774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [15624.865944] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:33:19 (1788168799) [15631.861968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15632.290415] LustreError: 521373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15632.306408] LustreError: 521373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15632.376635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15637.473589] LustreError: 521374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15637.667249] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15642.593791] LustreError: 521373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15647.093213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15647.381372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15652.163395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15659.691051] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15665.604582] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15672.242151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15672.532137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15678.706390] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15686.740432] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15692.741447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15692.881749] Lustre: lustre-OST0001: new disk, initializing [15692.884924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15692.889054] Lustre: 524210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15692.958965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15698.131968] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15701.550437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15701.564804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15701.649848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15705.681424] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15712.249430] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15712.262485] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15712.274534] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [15712.279741] Lustre: Skipped 1 previous similar message [15712.294939] Lustre: Skipped 1 previous similar message [15715.410915] Lustre: server umount lustre-OST0001 complete [15717.371406] LustreError: 523195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15717.408119] LustreError: 523195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15722.466292] LustreError: 523195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15722.509598] LustreError: 523195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15732.705932] LustreError: 523194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15732.727229] LustreError: 523194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15734.244447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15734.252312] LustreError: Skipped 1 previous similar message [15734.254558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15734.263062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15734.268955] Lustre: Skipped 1 previous similar message [15739.266169] Lustre: server umount lustre-OST0000 complete [15744.485310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15744.499987] LustreError: Skipped 1 previous similar message [15744.508096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15744.530341] Lustre: Skipped 1 previous similar message [15744.535534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15744.541426] Lustre: Skipped 3 previous similar messages [15749.801110] Lustre: server umount lustre-MDT0000 complete [15753.192087] LustreError: 521370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15753.209683] LustreError: 521370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15754.268192] LustreError: 521355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168931 with bad export cookie 8196802928302916588 [15754.275714] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15754.277948] LustreError: 521355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15760.645514] Lustre: server umount lustre-MDT0001 complete [15772.252372] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15774.969074] Key type lgssc unregistered [15775.192667] LNet: 526200:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15775.205774] LNetError: 526200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15775.220162] LNet: Removed LNI 192.168.204.120@tcp [15775.999770] Key type .llcrypt unregistered [15776.002470] Key type ._llcrypt unregistered [15835.404307] Key type ._llcrypt registered [15835.407897] Key type .llcrypt registered [15835.506790] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15836.920873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15836.950247] alg: No test for adler32 (adler32-zlib) [15838.171680] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15838.501098] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15840.327228] Key type lgssc registered [15841.735165] Lustre: Echo OBD driver; http://www.lustre.org/ [15852.495083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15852.540569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15853.814953] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15853.841692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15854.011427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15860.383758] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15870.585803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15870.675834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15870.729848] Lustre: 529001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15871.023283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15876.123755] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15884.955234] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15891.500333] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15896.546863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15896.558911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15896.579141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15897.059672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15897.081387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15897.088576] Lustre: Skipped 1 previous similar message [15900.800388] Lustre: server umount lustre-MDT0000 complete [15904.993050] LustreError: 529742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169081 with bad export cookie 17860015086525825556 [15904.998563] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15905.019550] LustreError: 529742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15905.501803] Lustre: server umount lustre-MDT0001 complete [15917.069075] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:38:11 (1788169091) [15924.441545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15924.940577] LustreError: 531513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15924.967731] LustreError: 531513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15925.033768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15930.345550] LustreError: 531514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15930.365306] LustreError: 531514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15930.610311] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15935.469168] LustreError: 531513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15940.582272] LustreError: 531514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15940.845417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15941.200601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15946.534816] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15954.854881] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15961.864927] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15969.229607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15969.456138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15969.722471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15977.258436] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15986.462935] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15996.342108] Lustre: Permanently deactivating lustre-OST0000 [15996.356384] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [16004.626853] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16005.730225] LustreError: 534505:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [16005.740586] LustreError: 534505:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16005.745288] LustreError: 534505:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16008.637529] LustreError: 534553:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16008.654533] LustreError: 534553:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16016.997420] Lustre: server umount lustre-OST0000 complete [16020.496969] LustreError: 534803:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16020.500983] LustreError: 534803:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16026.592825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16026.601288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16026.622021] Lustre: Skipped 1 previous similar message [16031.715165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16031.726523] Lustre: Skipped 2 previous similar messages [16032.812555] Lustre: server umount lustre-MDT0000 complete [16036.832243] LustreError: 532460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16036.865522] LustreError: 532460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16037.406691] LustreError: 531494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169214 with bad export cookie 17860015086525826144 [16037.411852] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16037.433912] LustreError: 531494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16047.768470] Lustre: server umount lustre-MDT0001 complete [16058.035156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16063.213215] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16066.542921] LustreError: 535964:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [16073.916551] Lustre: 536303:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.120@tcp [16076.602496] Lustre: server umount MGS complete [16088.828863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16089.404964] LustreError: 537027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16089.438088] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16089.449982] Lustre: Skipped 1 previous similar message [16089.520386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16094.041853] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16103.399814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16103.779858] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16108.733928] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16117.447635] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16123.539624] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16126.079085] Lustre: Permanently reactivating lustre-OST0000 [16126.093717] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [16126.099429] Lustre: Skipped 2 previous similar messages [16152.031286] LustreError: 538759:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [16159.216165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16159.544436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16159.546686] Lustre: Skipped 1 previous similar message [16165.537601] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16174.616849] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16182.241672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16182.249486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16182.251640] LustreError: Skipped 1 previous similar message [16182.258652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16182.273975] Lustre: Skipped 1 previous similar message [16182.293974] Lustre: Skipped 1 previous similar message [16182.761757] Lustre: server umount lustre-OST0000 complete [16187.361562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16187.375584] Lustre: Skipped 1 previous similar message [16192.483139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16192.489144] Lustre: Skipped 2 previous similar messages [16201.183227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16201.462432] Lustre: server umount lustre-MDT0000 complete [16202.728465] LustreError: 537755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16202.769736] LustreError: 537755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16206.246739] LustreError: 537008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169383 with bad export cookie 17860015086525827187 [16206.259131] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16206.264319] LustreError: 537008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16206.729649] Lustre: server umount lustre-MDT0001 complete [16220.332457] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16223.281499] Key type lgssc unregistered [16223.720942] LNet: 541296:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16223.730825] LNetError: 541296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16223.750057] LNet: Removed LNI 192.168.204.120@tcp [16224.864249] Key type .llcrypt unregistered [16224.867543] Key type ._llcrypt unregistered [16246.456341] Key type ._llcrypt registered [16246.458725] Key type .llcrypt registered [16246.610076] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [16264.613303] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16266.039039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16266.085987] alg: No test for adler32 (adler32-zlib) [16267.572813] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16268.000127] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16269.695190] Key type lgssc registered [16271.130261] Lustre: Echo OBD driver; http://www.lustre.org/ [16312.445421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16312.472665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16313.779842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16313.802607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16313.861808] Lustre: lustre-MDT0000: new disk, initializing [16313.933244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16313.948663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16319.727253] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16331.603379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16331.736721] Lustre: 544462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16331.791969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16331.796427] Lustre: Skipped 1 previous similar message [16331.891365] Lustre: lustre-MDT0001: new disk, initializing [16332.031934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16332.077955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16332.085434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16337.169239] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16348.118558] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16354.667137] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16361.288090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16361.558506] Lustre: lustre-OST0000: new disk, initializing [16361.564131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16361.574368] Lustre: 546432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16361.680862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16367.625295] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16368.205345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16368.217219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16368.411933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16378.986563] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16384.698868] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16384.963709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16390.507900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16390.839263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16394.227933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16394.234537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16394.268126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16394.279412] Lustre: Skipped 1 previous similar message [16399.330162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16399.360091] Lustre: Skipped 1 previous similar message [16399.705275] Lustre: server umount lustre-OST0000 complete [16409.063796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16409.069581] LustreError: Skipped 1 previous similar message [16409.072906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16409.081387] Lustre: Skipped 1 previous similar message [16409.084662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16414.692296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16414.699792] Lustre: Skipped 2 previous similar messages [16419.295142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16419.685792] Lustre: server umount lustre-MDT0000 complete [16424.621680] LustreError: 544456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169601 with bad export cookie 4667119000876787031 [16424.637304] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16424.639178] LustreError: 544456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16425.345223] Lustre: server umount lustre-MDT0001 complete [16435.768738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [16437.731809] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:46:52 (1788169612) [16446.534593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16446.932981] LustreError: 548898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16446.950042] LustreError: 548898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16446.991628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16452.074893] LustreError: 548899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16452.672979] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16456.164474] LustreError: 548898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16461.302778] LustreError: 548899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16462.155317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16462.404890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16466.625204] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16474.376398] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16480.924163] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16490.228647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16491.012104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16498.832862] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16509.020362] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16511.228025] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [16514.066381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [16522.731160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16522.734318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16522.745696] LustreError: Skipped 1 previous similar message [16522.754299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16522.788126] Lustre: Skipped 2 previous similar messages [16526.622269] Lustre: server umount lustre-OST0000 complete [16532.961931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16532.964646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16532.974757] Lustre: Skipped 1 previous similar message [16532.985125] Lustre: Skipped 2 previous similar messages [16536.805757] Lustre: server umount lustre-MDT0000 complete [16538.084820] LustreError: 550738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16538.101621] LustreError: 550738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16540.484672] LustreError: 548879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169717 with bad export cookie 4667119000876787913 [16540.489719] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16540.497333] LustreError: 548879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16541.036264] Lustre: server umount lustre-MDT0001 complete [16556.706756] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16562.712808] Key type lgssc unregistered [16563.430649] LNet: 552626:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16563.441624] LNetError: 552626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16563.481865] LNet: Removed LNI 192.168.204.120@tcp [16564.768165] Key type .llcrypt unregistered [16564.769801] Key type ._llcrypt unregistered [16590.275873] Key type ._llcrypt registered [16590.278499] Key type .llcrypt registered [16590.382977] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16590.774062] Key type .llcrypt unregistered [16590.782486] Key type ._llcrypt unregistered [16610.566630] Key type ._llcrypt registered [16610.570078] Key type .llcrypt registered [16610.727596] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16611.843917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16611.987651] alg: No test for adler32 (adler32-zlib) [16613.203858] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16613.475222] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16615.215225] Key type lgssc registered [16616.175388] Lustre: Echo OBD driver; http://www.lustre.org/ [16623.969374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16624.024518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16625.597379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16629.452103] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16635.873306] LustreError: 554840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16635.895102] LustreError: 554840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16636.431869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16636.641543] LustreError: 554841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16636.667126] LustreError: 554841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16636.669373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16642.037696] LustreError: 554840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16642.075534] LustreError: 554840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16642.700691] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16647.141291] LustreError: 554841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16647.168454] LustreError: 554841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16650.227650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16651.523146] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16655.336183] LustreError: 554835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16655.357975] LustreError: 554835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16659.680274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16660.104288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16665.158825] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16675.301437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16675.310293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16675.324417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16675.817566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16675.819556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16675.847751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16680.938772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16681.481946] Lustre: server umount lustre-OST0000 complete [16686.048556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16686.063112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16686.069823] Lustre: Skipped 1 previous similar message [16686.081197] Lustre: Skipped 1 previous similar message [16691.169184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16691.173827] Lustre: Skipped 2 previous similar messages [16691.903705] Lustre: server umount lustre-MDT0000 complete [16696.293664] LustreError: 556577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16696.322062] LustreError: 556577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16696.961850] LustreError: 554823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169873 with bad export cookie 15781395998693552287 [16696.964496] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16696.973353] LustreError: 554823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16697.530845] Lustre: server umount lustre-MDT0001 complete [16710.886578] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16714.952574] Key type lgssc unregistered [16715.325903] LNet: 557988:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16715.341659] LNetError: 557988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16715.365403] LNet: Removed LNI 192.168.204.120@tcp [16716.558585] Key type .llcrypt unregistered [16716.561846] Key type ._llcrypt unregistered [16749.029385] Key type ._llcrypt registered [16749.032803] Key type .llcrypt registered [16749.163793] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16750.560271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16750.606944] alg: No test for adler32 (adler32-zlib) [16751.801365] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16752.099199] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16753.719172] Key type lgssc registered [16754.952895] Lustre: Echo OBD driver; http://www.lustre.org/ [16764.135087] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16764.148633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16779.679460] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16784.799608] LustreError: 559380:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [16811.615269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16819.474041] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16829.436236] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16837.806844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16838.492389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16838.555125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16843.752935] LustreError: 559406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16843.756111] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16843.779567] LustreError: 559406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16846.838051] LustreError: 559405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16846.854313] LustreError: 559405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16851.951501] LustreError: 559406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16851.986488] LustreError: 559406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16855.426500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16855.974767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16861.214764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16869.947968] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16876.597846] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16884.706656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16884.720115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16884.744743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16886.760499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16886.760816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16886.760869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16886.784424] Lustre: Skipped 1 previous similar message [16890.805616] Lustre: server umount lustre-OST0000 complete [16896.997870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16897.003249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16897.015305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16897.026965] Lustre: Skipped 1 previous similar message [16901.106525] Lustre: server umount lustre-MDT0000 complete [16902.112476] LustreError: 560416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16902.129927] LustreError: 560416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16906.023790] LustreError: 559386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170082 with bad export cookie 6046023701197393467 [16906.049968] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16906.509851] Lustre: server umount lustre-MDT0001 complete [16921.283938] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16924.793849] Key type lgssc unregistered [16925.155161] LNet: 562955:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16925.166566] LNetError: 562955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16926.186789] LNet: Removed LNI 192.168.204.120@tcp [16927.013964] Key type .llcrypt unregistered [16927.016292] Key type ._llcrypt unregistered [16955.043550] Key type ._llcrypt registered [16955.050789] Key type .llcrypt registered [16955.184269] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16956.204668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16956.323914] alg: No test for adler32 (adler32-zlib) [16957.583330] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16957.814503] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16959.543249] Key type lgssc registered [16960.900789] Lustre: Echo OBD driver; http://www.lustre.org/ [16970.235467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16970.266870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16971.843907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16977.358504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16981.988324] LustreError: 564372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16982.001565] LustreError: 564372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16987.106435] LustreError: 564373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16987.632391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16988.294274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16993.482147] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17002.259202] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17009.317692] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17016.942686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17017.334296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17023.469092] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17031.608862] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17041.905458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [17047.008755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17047.011928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17047.028565] Lustre: Skipped 2 previous similar messages [17047.035995] Lustre: Skipped 2 previous similar messages [17047.877142] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [17051.363951] Lustre: server umount lustre-MDT0000 complete [17052.129618] LustreError: 564372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17052.144992] LustreError: 564372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17053.772199] LustreError: 564353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.20@tcp arrived at 1788170230 with bad export cookie 17227021906578611502 [17053.795124] LustreError: 564353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17055.731755] LustreError: 564354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170232 with bad export cookie 17227021906578610844 [17055.740236] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17055.753170] LustreError: 564354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17057.249166] LustreError: 564373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17057.252108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17057.275580] LustreError: 564373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17057.308480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17060.257575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17061.997633] Lustre: server umount lustre-OST0000 complete [17065.379524] LustreError: 564367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17065.391167] LustreError: 564367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17069.147530] Lustre: server umount lustre-MDT0001 complete [17082.325419] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [17085.731047] Key type lgssc unregistered [17086.022852] LNet: 568078:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17086.029700] LNetError: 568078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17086.055434] LNet: Removed LNI 192.168.204.120@tcp [17087.187906] Key type .llcrypt unregistered [17087.190241] Key type ._llcrypt unregistered [17116.377638] Key type ._llcrypt registered [17116.385241] Key type .llcrypt registered [17116.580124] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17117.938052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17117.980746] alg: No test for adler32 (adler32-zlib) [17119.214938] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17119.556751] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [17121.296688] Key type lgssc registered [17122.678309] Lustre: Echo OBD driver; http://www.lustre.org/ [17131.264182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17131.282504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17132.781505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17138.165288] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17142.242465] LustreError: 569497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17142.262905] LustreError: 569497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17147.367519] LustreError: 569498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17147.426597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17147.886050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17152.678461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17161.030956] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17167.761821] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17175.077503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17175.564690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17177.645778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [17182.359416] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17190.424660] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17198.064184] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17198.073325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17198.082964] Lustre: Skipped 1 previous similar message [17198.109285] Lustre: Skipped 1 previous similar message [17202.812641] Lustre: server umount lustre-MDT0001 complete [17203.171293] LustreError: 569494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17203.202985] LustreError: 569494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17208.292727] LustreError: 569497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17208.322088] LustreError: 569497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17213.411841] LustreError: 569498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17213.420645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17213.425036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17213.425664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17213.425668] Lustre: Skipped 1 previous similar message [17217.080117] Lustre: server umount lustre-OST0000 complete [17222.440426] Lustre: server umount lustre-MDT0000 complete [17240.627619] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [17243.919321] Key type lgssc unregistered [17244.204184] LNet: 573183:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17244.215878] LNetError: 573183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17244.243496] LNet: Removed LNI 192.168.204.120@tcp [17244.955626] Key type .llcrypt unregistered [17244.964502] Key type ._llcrypt unregistered [17272.781657] Key type ._llcrypt registered [17272.785299] Key type .llcrypt registered [17272.906688] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [17293.974324] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17294.676288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17294.766100] alg: No test for adler32 (adler32-zlib) [17295.993827] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17296.321308] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [17298.063399] Key type lgssc registered [17299.821255] Lustre: Echo OBD driver; http://www.lustre.org/ [17348.374912] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17367.190972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17367.239140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17368.554501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17368.613700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17368.760176] Lustre: lustre-MDT0000: new disk, initializing [17368.914444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17368.983125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17373.767828] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17391.899554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17392.173783] Lustre: lustre-OST0000: new disk, initializing [17392.177081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17392.182972] Lustre: Skipped 1 previous similar message [17392.187475] Lustre: 578237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17392.267953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17398.843216] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17399.928993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17399.938854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17399.979297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17408.965806] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17425.363300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17425.514222] Lustre: 577217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17425.572692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17425.691480] Lustre: lustre-MDT0001: new disk, initializing [17425.800224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17425.837615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17425.861078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17431.242909] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17441.258880] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17441.490946] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17445.551795] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17445.753671] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17450.980944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17450.990224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17450.991345] Lustre: Skipped 1 previous similar message [17450.993749] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17451.008575] Lustre: Skipped 1 previous similar message [17456.109143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17456.113746] Lustre: Skipped 2 previous similar messages [17456.978568] Lustre: server umount lustre-MDT0001 complete [17466.347787] LustreError: 579552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17466.370698] LustreError: 579552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17471.466990] LustreError: 579552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17471.494133] LustreError: 579552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17476.587212] LustreError: 577229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17476.588150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17476.611693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17476.615241] Lustre: Skipped 1 previous similar message [17480.451591] Lustre: server umount lustre-OST0000 complete [17481.703749] LustreError: 577228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17491.005936] Lustre: server umount lustre-MDT0000 complete [17506.290188] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [17509.236974] Key type lgssc unregistered [17509.626042] LNet: 581987:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17509.632911] LNetError: 581987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17509.652290] LNet: Removed LNI 192.168.204.120@tcp [17510.621616] Key type .llcrypt unregistered [17510.625411] Key type ._llcrypt unregistered [17554.854679] Key type ._llcrypt registered [17554.856149] Key type .llcrypt registered [17554.959794] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [17555.214466] Key type .llcrypt unregistered [17555.216528] Key type ._llcrypt unregistered [17573.634497] Key type ._llcrypt registered [17573.639984] Key type .llcrypt registered [17573.867970] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17575.776089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17575.804518] alg: No test for adler32 (adler32-zlib) [17577.119278] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17577.540873] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [17579.288192] Key type lgssc registered [17580.841927] Lustre: Echo OBD driver; http://www.lustre.org/ [17610.969186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17610.996577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17612.246169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17612.308406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17612.447344] Lustre: lustre-MDT0000: new disk, initializing [17612.611581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17612.630865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17617.402519] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17628.859781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17628.964794] Lustre: 585366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17629.017696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17629.023230] Lustre: Skipped 1 previous similar message [17629.113889] Lustre: lustre-MDT0001: new disk, initializing [17629.219039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17629.262692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17629.275809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17633.990739] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17644.564214] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17651.291893] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17657.909236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17658.192982] Lustre: lustre-OST0000: new disk, initializing [17658.203908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17658.214937] Lustre: 587332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17658.311437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17664.205876] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17665.625264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17665.643069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17665.783268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17673.614318] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17681.377362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17681.392132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17681.395481] Lustre: Skipped 1 previous similar message [17686.500533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17686.504100] Lustre: Skipped 4 previous similar messages [17687.415229] Lustre: server umount lustre-MDT0000 complete [17690.272489] LustreError: 585358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170867 with bad export cookie 3380608323483814864 [17690.274475] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17690.280157] LustreError: 585358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17690.527587] Lustre: server umount lustre-MDT0001 complete [17703.608423] Lustre: server umount lustre-OST0000 complete [17714.173510] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:08:08 (1788170888) [17721.783566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17722.221966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17727.460594] LustreError: 589364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17727.481502] LustreError: 589364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17727.567773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17732.582489] LustreError: 589363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17736.473580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17736.780676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17741.908152] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17750.179905] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17756.389317] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17776.713920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17776.947715] Lustre: Found index 0 for lustre-OST0000, updating log [17777.234456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17783.626676] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17791.825217] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17793.655202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17802.208321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17802.218430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17802.228203] Lustre: Skipped 1 previous similar message [17802.232837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17808.020560] Lustre: server umount lustre-OST0000 complete [17824.737307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17824.754037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17824.755592] Lustre: Skipped 2 previous similar messages [17824.775465] Lustre: Skipped 4 previous similar messages [17829.857601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17829.864150] Lustre: Skipped 1 previous similar message [17830.560942] Lustre: server umount lustre-MDT0000 complete [17830.883724] LustreError: 589364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17830.907445] LustreError: 589364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17834.463653] LustreError: 589345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171011 with bad export cookie 3380608323483816257 [17834.486178] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17834.777817] Lustre: server umount lustre-MDT0001 complete [17841.767286] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [17843.605049] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:10:18 (1788171018) [18461.798290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18462.119819] LustreError: 595775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18462.181845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18466.201691] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18467.298594] LustreError: 595776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18471.392253] LustreError: 595775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18474.097466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18474.430835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18478.567919] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18485.293468] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18491.613333] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18511.993524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18512.219636] Lustre: Found index 0 for lustre-OST0000, updating log [18512.515370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18518.280067] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18519.647408] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [18525.082526] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18527.394463] Lustre: server umount lustre-OST0000 complete [18530.785700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18530.792821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18530.810883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18531.876229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18531.876826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18531.895243] Lustre: Skipped 1 previous similar message [18533.991751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18534.000889] Lustre: Skipped 3 previous similar messages [18536.621719] Lustre: server umount lustre-MDT0000 complete [18537.057618] LustreError: 595775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18537.075421] LustreError: 595775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18540.105427] LustreError: 595757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171716 with bad export cookie 3380608323483817342 [18540.112461] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18540.115556] LustreError: 595757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [18540.541902] Lustre: server umount lustre-MDT0001 complete [18571.889994] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [18582.290925] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [18583.136289] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [18616.786388] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:23:11 (1788171791) [18634.181093] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [18636.856712] Key type lgssc unregistered [18637.109735] LNet: 602389:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18637.117427] LNetError: 602389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18765.160049] LNet: Removed LNI 192.168.204.120@tcp [18766.221804] Key type .llcrypt unregistered [18766.223896] Key type ._llcrypt unregistered [18782.228809] Key type ._llcrypt registered [18782.230785] Key type .llcrypt registered [18782.321923] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [18783.152216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18783.227122] alg: No test for adler32 (adler32-zlib) [18784.312735] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18784.530158] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [18786.175207] Key type lgssc registered [18787.191741] Lustre: Echo OBD driver; http://www.lustre.org/ [18791.361549] LNet: Added LNI 192.168.204.120@tcp100 [8/256/0/180] [18792.199623] LNet: Added LNI 192.168.204.120@tcp101 [8/256/0/180] [18793.144544] LNet: Added LNI 192.168.204.120@tcp102 [8/256/0/180] [18794.194970] LNet: Added LNI 192.168.204.120@tcp103 [8/256/0/180] [18795.135941] LNet: Added LNI 192.168.204.120@tcp104 [8/256/0/180] [18795.996278] LNet: Added LNI 192.168.204.120@tcp105 [8/256/0/180] [18796.890924] LNet: Added LNI 192.168.204.120@tcp106 [8/256/0/180] [18797.788378] LNet: Added LNI 192.168.204.120@tcp107 [8/256/0/180] [18798.675439] LNet: Added LNI 192.168.204.120@tcp108 [8/256/0/180] [18799.590981] LNet: Added LNI 192.168.204.120@tcp109 [8/256/0/180] [18800.495838] LNet: Added LNI 192.168.204.120@tcp110 [8/256/0/180] [18801.438821] LNet: Added LNI 192.168.204.120@tcp111 [8/256/0/180] [18802.323809] LNet: Added LNI 192.168.204.120@tcp112 [8/256/0/180] [18803.306015] LNet: Added LNI 192.168.204.120@tcp113 [8/256/0/180] [18804.380668] LNet: Added LNI 192.168.204.120@tcp114 [8/256/0/180] [18805.161560] LNet: Added LNI 192.168.204.120@tcp115 [8/256/0/180] [18806.076750] LNet: Added LNI 192.168.204.120@tcp116 [8/256/0/180] [18806.897769] LNet: Added LNI 192.168.204.120@tcp117 [8/256/0/180] [18807.820694] LNet: Added LNI 192.168.204.120@tcp118 [8/256/0/180] [18808.741659] LNet: Added LNI 192.168.204.120@tcp119 [8/256/0/180] [18809.647606] LNet: Added LNI 192.168.204.120@tcp120 [8/256/0/180] [18810.609245] LNet: Added LNI 192.168.204.120@tcp121 [8/256/0/180] [18811.485702] LNet: Added LNI 192.168.204.120@tcp122 [8/256/0/180] [18812.504705] LNet: Added LNI 192.168.204.120@tcp123 [8/256/0/180] [18813.444597] LNet: Added LNI 192.168.204.120@tcp124 [8/256/0/180] [18814.324642] LNet: Added LNI 192.168.204.120@tcp125 [8/256/0/180] [18815.129037] LNet: Added LNI 192.168.204.120@tcp126 [8/256/0/180] [18816.139826] LNet: Added LNI 192.168.204.120@tcp127 [8/256/0/180] [18816.996307] LNet: Added LNI 192.168.204.120@tcp128 [8/256/0/180] [18817.821703] LNet: Added LNI 192.168.204.120@tcp129 [8/256/0/180] [18818.687744] LNet: Added LNI 192.168.204.120@tcp130 [8/256/0/180] [18819.514801] LNet: Added LNI 192.168.204.120@tcp131 [8/256/0/180] [18820.584905] LNet: Added LNI 192.168.204.120@tcp132 [8/256/0/180] [18821.506482] LNet: Added LNI 192.168.204.120@tcp133 [8/256/0/180] [18822.319305] LNet: Added LNI 192.168.204.120@tcp134 [8/256/0/180] [18823.266475] LNet: Added LNI 192.168.204.120@tcp135 [8/256/0/180] [18829.899517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18829.945945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18831.058935] Lustre: 605425:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [18831.189323] LNetError: 605447:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18831.222269] LNetError: Protocol error connecting to 192.168.204.120@tcp119 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18831.251100] LNetError: Skipped 4 previous similar messages [18831.312272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18831.336409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18831.401977] Lustre: lustre-MDT0000: new disk, initializing [18831.503663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18831.521127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18831.648716] LNet: 6 local NIs in recovery (showing 6): 192.168.204.120@tcp124, 192.168.204.120@tcp126, 192.168.204.120@tcp127, 192.168.204.120@tcp128, 192.168.204.120@tcp129, 192.168.204.120@tcp119 [18832.289543] LNetError: 602884:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18832.319801] LNetError: 602884:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [18832.327812] LNetError: Protocol error connecting to 192.168.204.120@tcp107 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18832.338494] LNetError: Skipped 1 previous similar message [18834.784287] LNetError: 605447:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18834.798680] LNetError: 605447:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [18834.810884] LNetError: Protocol error connecting to 192.168.204.120@tcp124 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18834.838218] LNetError: Skipped 9 previous similar messages [18836.115374] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18838.881298] LNetError: 605459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18838.902368] LNetError: 605459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [18838.908935] LNetError: Protocol error connecting to 192.168.204.120@tcp124 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18846.058739] LNetError: 602887:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18846.075065] LNetError: Protocol error connecting to 192.168.204.120@tcp124 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18846.598898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18846.805374] Lustre: 605443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18846.869522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18846.873624] Lustre: Skipped 1 previous similar message [18846.942946] Lustre: lustre-MDT0001: new disk, initializing [18846.973718] Lustre: 606244:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [18846.980152] Lustre: 606244:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18847.086633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18847.132897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18847.149049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18851.847238] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18861.899076] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18862.373569] LNetError: 605453:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18862.390607] LNetError: Protocol error connecting to 192.168.204.120@tcp123 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18867.157851] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18872.660321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18872.933968] Lustre: lustre-OST0000: new disk, initializing [18872.942548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18872.949131] Lustre: 607416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18873.062568] Lustre: 607421:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [18873.068436] Lustre: 607421:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [18873.109812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18878.852800] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [18880.800602] LNetError: 605459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18880.830959] LNetError: 605459:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [18880.846173] LNetError: Protocol error connecting to 192.168.204.120@tcp129 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18880.856793] LNetError: Skipped 6 previous similar messages [18882.075215] Lustre: 607894:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [18882.081032] Lustre: 607894:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18882.130152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18882.143422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18882.255081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18888.233863] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18892.128626] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp124 [18897.897177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18897.899696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18897.901551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18897.903693] LustreError: Skipped 1 previous similar message [18897.932824] Lustre: Skipped 1 previous similar message [18901.765993] Lustre: server umount lustre-OST0000 complete [18908.130857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18908.144074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18908.152043] Lustre: Skipped 2 previous similar messages [18911.810209] Lustre: server umount lustre-MDT0000 complete [18913.570488] LNetError: 602886:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [18913.576025] LNetError: 602886:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [18913.582884] LNetError: Protocol error connecting to 192.168.204.120@tcp128 at host 192.168.204.120:988: is it running a compatible version of Lustre? [18913.598384] LNetError: Skipped 13 previous similar messages [18915.163130] LustreError: 605434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172092 with bad export cookie 16180098904313423755 [18915.169603] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18915.176754] LustreError: 605434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [18915.426876] Lustre: server umount lustre-MDT0001 complete [18926.769852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [18929.602444] Key type lgssc unregistered [18930.117390] LNet: 609327:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18930.151902] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18930.177300] LNet: Removed LNI 192.168.204.120@tcp [18931.231258] LNet: Removed LNI 192.168.204.120@tcp100 [18931.251545] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18931.270217] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18931.296589] LNet: Removed LNI 192.168.204.120@tcp101 [18932.319703] LNet: Removed LNI 192.168.204.120@tcp102 [18932.328844] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18932.362888] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18932.368307] LNet: Removed LNI 192.168.204.120@tcp103 [18933.409236] LNet: Removed LNI 192.168.204.120@tcp104 [18934.431979] LNet: Removed LNI 192.168.204.120@tcp105 [18934.435174] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18934.439983] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [18934.443089] LNet: Removed LNI 192.168.204.120@tcp106 [18935.455208] LNet: Removed LNI 192.168.204.120@tcp107 [18936.479435] LNet: Removed LNI 192.168.204.120@tcp108 [18937.503259] LNet: Removed LNI 192.168.204.120@tcp109 [18938.591222] LNet: Removed LNI 192.168.204.120@tcp110 [18938.595554] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18938.612421] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [18938.634964] LNet: Removed LNI 192.168.204.120@tcp111 [18939.680148] LNet: Removed LNI 192.168.204.120@tcp112 [18940.767912] LNet: Removed LNI 192.168.204.120@tcp113 [18941.791312] LNet: Removed LNI 192.168.204.120@tcp114 [18942.829232] LNet: Removed LNI 192.168.204.120@tcp115 [18943.841412] LNet: Removed LNI 192.168.204.120@tcp116 [18944.863163] LNet: Removed LNI 192.168.204.120@tcp117 [18945.887334] LNet: Removed LNI 192.168.204.120@tcp118 [18946.911277] LNet: Removed LNI 192.168.204.120@tcp119 [18946.917049] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18946.929483] LNetError: 609327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [18946.939872] LNet: Removed LNI 192.168.204.120@tcp120 [18947.999786] LNet: Removed LNI 192.168.204.120@tcp121 [18949.023841] LNet: Removed LNI 192.168.204.120@tcp122 [18950.047144] LNet: Removed LNI 192.168.204.120@tcp123 [18950.056722] LNet: Removed LNI 192.168.204.120@tcp124 [18951.071249] LNet: Removed LNI 192.168.204.120@tcp125 [18952.095210] LNet: Removed LNI 192.168.204.120@tcp126 [18953.119235] LNet: Removed LNI 192.168.204.120@tcp127 [18954.143491] LNet: Removed LNI 192.168.204.120@tcp128 [18955.167187] LNet: Removed LNI 192.168.204.120@tcp129 [18955.173520] LNet: Removed LNI 192.168.204.120@tcp130 [18955.178107] LNet: Removed LNI 192.168.204.120@tcp131 [18955.184775] LNet: Removed LNI 192.168.204.120@tcp132 [18955.191769] LNet: Removed LNI 192.168.204.120@tcp133 [18955.193745] LNet: Removed LNI 192.168.204.120@tcp134 [18955.207332] LNet: Removed LNI 192.168.204.120@tcp135 [18956.293325] Key type .llcrypt unregistered [18956.298261] Key type ._llcrypt unregistered [18985.957818] Key type ._llcrypt registered [18985.960212] Key type .llcrypt registered [18986.111280] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [19001.600170] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [19003.076023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19003.214083] alg: No test for adler32 (adler32-zlib) [19004.360462] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19004.570553] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [19006.280853] Key type lgssc registered [19007.887845] Lustre: Echo OBD driver; http://www.lustre.org/ [19050.693188] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:30:25 (1788172225) [19068.791926] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [19071.180638] Key type lgssc unregistered [19071.489659] LNet: 613421:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19071.501387] LNetError: 613421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19071.520785] LNet: Removed LNI 192.168.204.120@tcp [19072.410584] Key type .llcrypt unregistered [19072.415157] Key type ._llcrypt unregistered [19088.296820] Key type ._llcrypt registered [19088.303730] Key type .llcrypt registered [19088.461167] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [19089.972395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19090.000567] alg: No test for adler32 (adler32-zlib) [19091.137839] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19091.561955] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [19093.343215] Key type lgssc registered [19094.732752] Lustre: Echo OBD driver; http://www.lustre.org/ [19104.343476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19104.360357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19105.546749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19105.569521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19105.611807] Lustre: lustre-MDT0000: new disk, initializing [19105.684247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19105.716769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19110.319610] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19124.674058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19124.854294] Lustre: 614633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19124.988393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19125.002467] Lustre: Skipped 1 previous similar message [19125.252408] Lustre: lustre-MDT0001: new disk, initializing [19125.479416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19125.579205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19125.605814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19131.729151] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19142.597436] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19149.213678] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19157.149910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19157.484141] Lustre: lustre-OST0000: new disk, initializing [19157.488542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19157.494368] Lustre: 616605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19157.591164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19164.713827] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19165.841993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19165.861834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19166.001480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19176.290567] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19181.039783] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [19187.023935] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19187.057143] LNet: Added LNI 192.168.204.120@tcp10 [8/256/0/180] [19193.527319] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19193.563851] LNet: Added LNI 192.168.204.120@tcp11 [8/256/0/180] [19198.137399] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19198.188960] LNet: Added LNI 192.168.204.120@tcp12 [8/256/0/180] [19203.620626] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19203.652229] LNet: Added LNI 192.168.204.120@tcp13 [8/256/0/180] [19208.443906] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19208.494894] LNet: Added LNI 192.168.204.120@tcp14 [8/256/0/180] [19213.404381] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19213.427144] LNet: Added LNI 192.168.204.120@tcp15 [8/256/0/180] [19218.654376] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19218.688975] LNet: Added LNI 192.168.204.120@tcp16 [8/256/0/180] [19224.114432] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19224.146680] LNet: Added LNI 192.168.204.120@tcp17 [8/256/0/180] [19229.479949] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19229.513160] LNet: Added LNI 192.168.204.120@tcp18 [8/256/0/180] [19233.901559] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19233.929961] LNet: Added LNI 192.168.204.120@tcp19 [8/256/0/180] [19238.994027] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19239.033285] LNet: Added LNI 192.168.204.120@tcp20 [8/256/0/180] [19244.831923] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19244.864828] LNet: Added LNI 192.168.204.120@tcp21 [8/256/0/180] [19250.950900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19251.007746] LNet: Added LNI 192.168.204.120@tcp22 [8/256/0/180] [19258.195438] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19258.261738] LNet: Added LNI 192.168.204.120@tcp23 [8/256/0/180] [19265.147610] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19265.175308] LNet: Added LNI 192.168.204.120@tcp24 [8/256/0/180] [19271.762439] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19271.805464] LNet: Added LNI 192.168.204.120@tcp25 [8/256/0/180] [19278.344885] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19278.377875] LNet: Added LNI 192.168.204.120@tcp26 [8/256/0/180] [19284.859746] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19284.921501] LNet: Added LNI 192.168.204.120@tcp27 [8/256/0/180] [19291.141242] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19291.164271] LNet: Added LNI 192.168.204.120@tcp28 [8/256/0/180] [19297.713867] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19297.774962] LNet: Added LNI 192.168.204.120@tcp29 [8/256/0/180] [19306.248517] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19306.273607] LNet: Added LNI 192.168.204.120@tcp30 [8/256/0/180] [19314.923507] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19314.960502] LNet: Added LNI 192.168.204.120@tcp31 [8/256/0/180] [19322.772255] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19322.819086] LNet: Added LNI 192.168.204.120@tcp32 [8/256/0/180] [19329.892730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19329.926341] LNet: Added LNI 192.168.204.120@tcp33 [8/256/0/180] [19336.975853] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19337.009989] LNet: Added LNI 192.168.204.120@tcp34 [8/256/0/180] [19344.369902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19344.391415] LNet: Added LNI 192.168.204.120@tcp35 [8/256/0/180] [19351.542788] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19351.589690] LNet: Added LNI 192.168.204.120@tcp36 [8/256/0/180] [19357.619561] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19357.669978] LNet: Added LNI 192.168.204.120@tcp37 [8/256/0/180] [19364.253936] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19364.278341] LNet: Added LNI 192.168.204.120@tcp38 [8/256/0/180] [19372.363482] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19372.399757] LNet: Added LNI 192.168.204.120@tcp39 [8/256/0/180] [19379.437542] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19379.469833] LNet: Added LNI 192.168.204.120@tcp40 [8/256/0/180] [19386.828657] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19386.875801] LNet: Added LNI 192.168.204.120@tcp41 [8/256/0/180] [19393.791970] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19393.813104] LNet: Added LNI 192.168.204.120@tcp42 [8/256/0/180] [19401.575502] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19401.609955] LNet: Added LNI 192.168.204.120@tcp43 [8/256/0/180] [19409.303875] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19409.333496] LNet: Added LNI 192.168.204.120@tcp44 [8/256/0/180] [19417.114382] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19417.153858] LNet: Added LNI 192.168.204.120@tcp45 [8/256/0/180] [19424.626568] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19424.656443] LNet: Added LNI 192.168.204.120@tcp46 [8/256/0/180] [19433.208550] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19433.252161] LNet: Added LNI 192.168.204.120@tcp47 [8/256/0/180] [19441.741731] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19441.772892] LNet: Added LNI 192.168.204.120@tcp48 [8/256/0/180] [19450.361256] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19450.400802] LNet: Added LNI 192.168.204.120@tcp49 [8/256/0/180] [19460.648773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19460.707567] LNet: Added LNI 192.168.204.120@tcp50 [8/256/0/180] [19483.624085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19483.625290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19483.659261] Lustre: Skipped 2 previous similar messages [19483.672905] Lustre: Skipped 2 previous similar messages [19486.026520] Lustre: server umount lustre-MDT0000 complete [19490.865730] LustreError: 614627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172667 with bad export cookie 15537835448323358666 [19490.882268] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19490.883831] LustreError: 614627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19491.320028] Lustre: server umount lustre-MDT0001 complete [19509.222961] Lustre: 613905:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172670/real 1788172670] req@00000000fd0b5ef1 x1875034524267264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19509.240393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19510.840460] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [19512.096578] Lustre: 613906:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172672/real 1788172672] req@00000000b4e10acc x1875034524267392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172688 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19514.335414] Lustre: 613907:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172675/real 1788172675] req@000000000693824f x1875034524267520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19516.390880] Lustre: 613908:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172677/real 1788172677] req@00000000080a6013 x1875034524267776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19521.215533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19521.503310] Lustre: 613905:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172682/real 1788172682] req@000000000a39ee2f x1875034524267904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19535.904400] LustreError: 613904:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000025cf4d52 x1875034524268672/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19536.220690] LustreError: 616597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19536.240092] LustreError: 616597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [19536.319386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19538.400813] LustreError: 616598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19538.405246] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19539.433971] LustreError: 616597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19541.590574] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19544.558155] LustreError: 616598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19549.669298] LustreError: 616597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19551.096680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19551.471392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19554.559064] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19555.626919] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19564.314564] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19570.148538] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19592.672903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19592.690857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19592.703963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19592.713348] Lustre: Skipped 1 previous similar message [19595.760535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19595.778294] Lustre: Skipped 1 previous similar message [19597.888444] Lustre: server umount lustre-OST0000 complete [19602.915117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19602.925603] LustreError: Skipped 1 previous similar message [19602.938476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19602.954751] Lustre: Skipped 1 previous similar message [19602.964846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19608.883174] Lustre: server umount lustre-MDT0000 complete [19611.106954] LustreError: 642144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19611.140327] LustreError: 642144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19613.080329] LustreError: 614625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172789 with bad export cookie 15537835448323361284 [19613.087095] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19613.093294] LustreError: 614625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19613.462417] Lustre: server umount lustre-MDT0001 complete [19625.737453] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [19629.218610] Key type lgssc unregistered [19629.589850] LNet: 643650:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19629.600672] LNetError: 643650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19629.606987] LNet: Removed LNI 192.168.204.120@tcp [19629.609680] LNet: Removed LNI 192.168.204.120@tcp10 [19629.611625] LNet: Removed LNI 192.168.204.120@tcp11 [19629.615462] LNet: Removed LNI 192.168.204.120@tcp12 [19629.618199] LNet: Removed LNI 192.168.204.120@tcp13 [19629.620429] LNet: Removed LNI 192.168.204.120@tcp14 [19629.623479] LNet: Removed LNI 192.168.204.120@tcp15 [19629.629076] LNet: Removed LNI 192.168.204.120@tcp16 [19629.631732] LNet: Removed LNI 192.168.204.120@tcp17 [19629.634335] LNet: Removed LNI 192.168.204.120@tcp18 [19629.636593] LNet: Removed LNI 192.168.204.120@tcp19 [19629.638967] LNet: Removed LNI 192.168.204.120@tcp20 [19629.641033] LNet: Removed LNI 192.168.204.120@tcp21 [19629.645070] LNet: Removed LNI 192.168.204.120@tcp22 [19629.648964] LNet: Removed LNI 192.168.204.120@tcp23 [19629.652974] LNet: Removed LNI 192.168.204.120@tcp24 [19629.656300] LNet: Removed LNI 192.168.204.120@tcp25 [19629.661582] LNet: Removed LNI 192.168.204.120@tcp26 [19629.664506] LNet: Removed LNI 192.168.204.120@tcp27 [19629.668438] LNet: Removed LNI 192.168.204.120@tcp28 [19629.672851] LNet: Removed LNI 192.168.204.120@tcp29 [19629.679848] LNet: Removed LNI 192.168.204.120@tcp30 [19629.683896] LNet: Removed LNI 192.168.204.120@tcp31 [19629.691246] LNet: Removed LNI 192.168.204.120@tcp32 [19629.694278] LNet: Removed LNI 192.168.204.120@tcp33 [19629.698464] LNet: Removed LNI 192.168.204.120@tcp34 [19629.705246] LNet: Removed LNI 192.168.204.120@tcp35 [19629.708707] LNet: Removed LNI 192.168.204.120@tcp36 [19629.711657] LNet: Removed LNI 192.168.204.120@tcp37 [19629.715711] LNet: Removed LNI 192.168.204.120@tcp38 [19629.719245] LNet: Removed LNI 192.168.204.120@tcp39 [19629.723157] LNet: Removed LNI 192.168.204.120@tcp40 [19629.726672] LNet: Removed LNI 192.168.204.120@tcp41 [19629.729231] LNet: Removed LNI 192.168.204.120@tcp42 [19629.732453] LNet: Removed LNI 192.168.204.120@tcp43 [19629.734809] LNet: Removed LNI 192.168.204.120@tcp44 [19629.740981] LNet: Removed LNI 192.168.204.120@tcp45 [19629.743935] LNet: Removed LNI 192.168.204.120@tcp46 [19629.746653] LNet: Removed LNI 192.168.204.120@tcp47 [19629.748983] LNet: Removed LNI 192.168.204.120@tcp48 [19629.751882] LNet: Removed LNI 192.168.204.120@tcp49 [19629.764578] LNet: Removed LNI 192.168.204.120@tcp50 [19630.573665] Key type .llcrypt unregistered [19630.575344] Key type ._llcrypt unregistered [19652.628642] Key type ._llcrypt registered [19652.633952] Key type .llcrypt registered [19652.748664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [19667.429222] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [19668.364535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19668.389951] alg: No test for adler32 (adler32-zlib) [19669.429935] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19669.710380] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [19671.503138] Key type lgssc registered [19672.794628] Lustre: Echo OBD driver; http://www.lustre.org/ [19709.463211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19709.475370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19710.813681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19710.868326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19710.986207] Lustre: lustre-MDT0000: new disk, initializing [19711.061159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19711.074878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19716.167533] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19730.026149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19730.138144] Lustre: 646850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19730.189585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19730.192949] Lustre: Skipped 1 previous similar message [19730.257478] Lustre: lustre-MDT0001: new disk, initializing [19730.342977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19730.383350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19730.391750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19735.845355] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19745.668237] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19751.456373] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19759.651669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19759.875789] Lustre: *** cfs_fail_loc=90f, val=0*** [19760.009595] Lustre: lustre-OST0000: new disk, initializing [19760.018248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19760.039131] Lustre: 648866:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19760.172635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19766.506716] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19769.931177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19769.950070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19770.112493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19777.181426] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19785.700696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19785.709117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19785.720883] LustreError: Skipped 1 previous similar message [19785.723357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19785.732014] Lustre: Skipped 1 previous similar message [19785.739467] Lustre: Skipped 1 previous similar message [19790.818750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19790.829444] Lustre: Skipped 1 previous similar message [19795.951970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19795.957683] Lustre: Skipped 1 previous similar message [19799.008034] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [19799.235311] Lustre: server umount lustre-OST0000 complete [19806.185530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19806.194706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19806.212024] Lustre: Skipped 1 previous similar message [19806.229780] Lustre: Skipped 1 previous similar message [19810.087804] Lustre: server umount lustre-MDT0000 complete [19814.800681] LustreError: 646842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172991 with bad export cookie 11144503698774729021 [19814.828940] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19815.221835] Lustre: server umount lustre-MDT0001 complete [19827.542641] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [19830.818951] Key type lgssc unregistered [19831.167713] LNet: 650777:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19831.174541] LNetError: 650777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19831.187153] LNet: Removed LNI 192.168.204.120@tcp [19832.370990] Key type .llcrypt unregistered [19832.373118] Key type ._llcrypt unregistered [19854.166503] Key type ._llcrypt registered [19854.168268] Key type .llcrypt registered [19854.263323] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [19855.495205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19855.509712] alg: No test for adler32 (adler32-zlib) [19856.549760] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19856.826616] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [19858.503356] Key type lgssc registered [19859.738814] Lustre: Echo OBD driver; http://www.lustre.org/ [19867.870862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19867.885707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19869.347565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19874.183574] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19879.905634] LustreError: 652051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19879.917390] LustreError: 652051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19882.849660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19883.283144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19887.605088] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19895.242857] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19900.889722] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19908.628400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19909.049076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19916.764750] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19924.966498] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19936.329327] loop0: detected capacity change from 0 to 204800000 [19936.601254] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19936.609657] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19936.630849] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19942.939386] loop0: detected capacity change from 0 to 204800000 [19943.052510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19943.063116] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19943.070817] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19948.761466] loop0: detected capacity change from 0 to 204800000 [19955.465500] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19955.708972] Lustre: fs15246-OST0000: new disk, initializing [19955.722120] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [19955.732236] Lustre: 655647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19955.829157] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19963.130234] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19973.393232] loop1: detected capacity change from 0 to 204800000 [19980.261279] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19980.510807] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [19980.629303] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [19980.790885] Lustre: fs15246-MDT0000: new disk, initializing [19980.986780] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19981.025841] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19981.173703] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [19982.241098] Lustre: 656768:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d407d34d x1875035326300032/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788173175 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [19986.479414] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [19987.337144] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [19987.356587] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [19987.413673] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20012.026712] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20012.056753] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20014.634814] Lustre: server umount fs15246-MDT0000 complete [20017.125640] LustreError: 652048:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20022.259789] LustreError: 654100:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20024.151159] Lustre: server umount fs15246-OST0000 complete [20037.098871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20037.110431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20037.132607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20041.368878] Lustre: server umount lustre-OST0000 complete [20047.335583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20047.350208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20047.360091] Lustre: Skipped 1 previous similar message [20047.362875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20047.375913] Lustre: Skipped 1 previous similar message [20051.912672] Lustre: server umount lustre-MDT0000 complete [20052.962634] LustreError: 652780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20052.978086] LustreError: 652780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20056.005141] LustreError: 652032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173232 with bad export cookie 18073981929406183667 [20056.012662] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20056.033620] LustreError: 652032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20056.459648] Lustre: server umount lustre-MDT0001 complete [20070.400465] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [20073.789919] Key type lgssc unregistered [20074.219664] LNet: 659865:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20074.233435] LNetError: 659865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20074.251782] LNet: Removed LNI 192.168.204.120@tcp [20075.614133] Key type .llcrypt unregistered [20075.616229] Key type ._llcrypt unregistered [20096.193387] Key type ._llcrypt registered [20096.196288] Key type .llcrypt registered [20096.320812] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [20097.298620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20097.443267] alg: No test for adler32 (adler32-zlib) [20098.578157] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20098.823636] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [20100.503976] Key type lgssc registered [20101.711041] Lustre: Echo OBD driver; http://www.lustre.org/ [20137.364391] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [20146.063109] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [20181.584198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20181.610808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20182.854029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20182.895841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20182.978737] Lustre: lustre-MDT0000: new disk, initializing [20183.073676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20183.088898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20187.399512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [20197.999909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20198.133402] Lustre: 664336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20198.200789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20198.206399] Lustre: Skipped 1 previous similar message [20198.275851] Lustre: lustre-MDT0001: new disk, initializing [20198.397272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20198.441197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20198.456758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20203.589803] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [20212.920570] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20217.421619] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20223.633427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20223.976415] Lustre: lustre-OST0000: new disk, initializing [20223.980473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20223.989453] Lustre: 666307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20224.081107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20229.939093] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [20233.820869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20233.835744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20233.882080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20238.605824] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20243.828275] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20244.082435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20249.392695] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20249.695742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20254.695951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20254.700535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20254.708116] LustreError: Skipped 1 previous similar message [20254.735940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20254.747591] Lustre: Skipped 1 previous similar message [20257.727692] Lustre: server umount lustre-OST0000 complete [20264.930871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20264.933602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20264.952661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20264.978469] Lustre: Skipped 2 previous similar messages [20264.999404] Lustre: Skipped 1 previous similar message [20267.300248] Lustre: server umount lustre-MDT0000 complete [20270.785299] LustreError: 664328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173447 with bad export cookie 614625178623839987 [20270.788164] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20270.797381] LustreError: 664328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [20271.202821] Lustre: server umount lustre-MDT0001 complete [20297.273941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [20307.821911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [20337.766257] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20033 sec ========= 06:51:52 (1788173512) [20339.285739] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:51:54 (1788173514) === [20342.472255] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:51:57 (1788173517) === [20380.417186] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [20383.151495] Key type lgssc unregistered [20383.399267] LNet: 672951:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20383.407718] LNetError: 672951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20383.430673] LNet: Removed LNI 192.168.204.120@tcp [20384.431945] Key type .llcrypt unregistered [20384.434237] Key type ._llcrypt unregistered