[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494325833 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003182] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008968] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011102] LSM: Security Framework initializing [ 0.012035] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.015034] *** VALIDATE selinux *** [ 0.023208] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028548] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031116] *** VALIDATE tmpfs *** [ 0.032486] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037044] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.043951] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.046154] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055288] rcu: Hierarchical SRCU implementation. [ 0.057431] smp: Bringing up secondary CPUs ... [ 0.058630] x86: Booting SMP configuration: [ 0.059022] .... node #0, CPUs: #1 #2 #3 [ 0.062515] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144700] node 0 deferred pages initialised in 75ms [ 0.147263] devtmpfs: initialized [ 0.149317] x86/mm: Memory block size: 128MB [ 0.153951] gcov: version magic: 0x41383552 [ 0.158287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165404] pinctrl core: initialized pinctrl subsystem [ 0.168198] [ 0.168680] ************************************************************* [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172017] ** ** [ 0.174013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177013] ** ** [ 0.179011] ** This means that this kernel is built to expose internal ** [ 0.181014] ** IOMMU data structures, which may compromise security on ** [ 0.184016] ** your system. ** [ 0.186012] ** ** [ 0.188016] ** If you see this message and you are not debugging the ** [ 0.191014] ** kernel, report this immediately to your vendor! ** [ 0.193018] ** ** [ 0.196017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198013] ************************************************************* [ 0.203204] NET: Registered protocol family 16 [ 0.205491] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214451] cpuidle: using governor menu [ 0.215773] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217521] PCI: Using configuration type 1 for base access [ 0.219162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232101] cryptd: max_cpu_qlen set to 1000 [ 0.234392] ACPI: Added _OSI(Module Device) [ 0.236034] ACPI: Added _OSI(Processor Device) [ 0.238017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245641] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253572] ACPI: Interpreter enabled [ 0.255061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257013] ACPI: Using IOAPIC for interrupt routing [ 0.259177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264169] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272755] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284000] acpiphp: Slot [2] registered [ 0.284000] acpiphp: Slot [5] registered [ 0.287164] acpiphp: Slot [6] registered [ 0.288108] acpiphp: Slot [7] registered [ 0.290148] acpiphp: Slot [8] registered [ 0.292169] acpiphp: Slot [9] registered [ 0.294090] acpiphp: Slot [10] registered [ 0.296132] acpiphp: Slot [3] registered [ 0.297101] acpiphp: Slot [4] registered [ 0.299161] acpiphp: Slot [11] registered [ 0.300127] acpiphp: Slot [12] registered [ 0.301144] acpiphp: Slot [13] registered [ 0.303109] acpiphp: Slot [14] registered [ 0.305128] acpiphp: Slot [15] registered [ 0.306149] acpiphp: Slot [16] registered [ 0.308195] acpiphp: Slot [17] registered [ 0.310109] acpiphp: Slot [18] registered [ 0.311078] acpiphp: Slot [19] registered [ 0.312120] acpiphp: Slot [20] registered [ 0.314118] acpiphp: Slot [21] registered [ 0.315106] acpiphp: Slot [22] registered [ 0.317084] acpiphp: Slot [23] registered [ 0.319134] acpiphp: Slot [24] registered [ 0.320113] acpiphp: Slot [25] registered [ 0.322113] acpiphp: Slot [26] registered [ 0.324121] acpiphp: Slot [27] registered [ 0.326215] acpiphp: Slot [28] registered [ 0.327123] acpiphp: Slot [29] registered [ 0.328079] acpiphp: Slot [30] registered [ 0.330106] acpiphp: Slot [31] registered [ 0.331104] PCI host bridge to bus 0000:00 [ 0.333030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353361] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363714] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368385] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378740] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381899] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387949] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471952] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.498021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.534017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549889] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.587015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600374] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.619023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.646019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.658182] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666940] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.716933] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.739015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.773024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.787853] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.790419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.793431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.797485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.800254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.806111] iommu: Default domain type: Passthrough [ 0.807492] SCSI subsystem initialized [ 0.808207] ACPI: bus type USB registered [ 0.810118] usbcore: registered new interface driver usbfs [ 0.812122] usbcore: registered new interface driver hub [ 0.814115] usbcore: registered new device driver usb [ 0.816163] pps_core: LinuxPPS API ver. 1 registered [ 0.818012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.821080] PTP clock support registered [ 0.824085] EDAC MC: Ver: 3.0.0 [ 0.825519] PCI: Using ACPI for IRQ routing [ 0.826933] NetLabel: Initializing [ 0.827011] NetLabel: domain hash size = 128 [ 0.828009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829187] NetLabel: unlabeled traffic allowed by default [ 0.831442] vgaarb: loaded [ 0.833390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.840402] clocksource: Switched to clocksource kvm-clock [ 0.944348] VFS: Disk quotas dquot_6.6.0 [ 0.945972] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.948340] *** VALIDATE ramfs *** [ 0.949719] *** VALIDATE hugetlbfs *** [ 0.950735] pnp: PnP ACPI init [ 0.952255] pnp: PnP ACPI: found 6 devices [ 0.970026] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.973942] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.976480] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.978928] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.982108] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.984311] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.987288] NET: Registered protocol family 2 [ 0.989830] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.994870] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999298] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005482] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.010092] TCP: Hash tables configured (established 65536 bind 65536) [ 1.014169] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.017685] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.020828] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024082] NET: Registered protocol family 1 [ 1.028324] RPC: Registered named UNIX socket transport module. [ 1.031087] RPC: Registered udp transport module. [ 1.033119] RPC: Registered tcp transport module. [ 1.035449] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038905] NET: Registered protocol family 44 [ 1.041563] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044297] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046862] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049660] PCI: CLS 0 bytes, default 64 [ 1.051683] Unpacking initramfs... [ 2.487738] debug: unmapping init [mem 0xffff97afbcc54000-0xffff97afbffbffff] [ 2.496377] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.499075] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.501834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.999209] Initialise system trusted keyrings [ 3.001092] Key type blacklist registered [ 3.003791] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.015703] zbud: loaded [ 3.019400] *** VALIDATE nfs *** [ 3.020829] *** VALIDATE nfs4 *** [ 3.022749] pstore: using deflate compression [ 3.026203] Platform Keyring initialized [ 3.133278] NET: Registered protocol family 38 [ 3.135504] Key type asymmetric registered [ 3.138108] Asymmetric key parser 'x509' registered [ 3.141032] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.144251] io scheduler mq-deadline registered [ 3.145905] io scheduler kyber registered [ 3.147481] io scheduler bfq registered [ 3.149133] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.152453] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.155414] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.158501] ACPI: Power Button [PWRF] [ 3.164569] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.171676] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.184849] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.195282] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.210476] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.240175] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.268853] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.275657] Non-volatile memory driver v1.3 [ 3.277580] Linux agpgart interface v0.103 [ 3.312701] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.316389] vda: detected capacity change from 0 to 74326016 [ 3.335568] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.339164] vdb: detected capacity change from 0 to 1073741824 [ 3.357152] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360113] vdc: detected capacity change from 0 to 2621440000 [ 3.374236] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.377167] vdd: detected capacity change from 0 to 2621440000 [ 3.392137] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396096] vde: detected capacity change from 0 to 4294967296 [ 3.412680] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415962] vdf: detected capacity change from 0 to 4294967296 [ 3.423444] libphy: Fixed MDIO Bus: probed [ 3.430091] usbcore: registered new interface driver usbserial_generic [ 3.432349] usbserial: USB Serial support registered for generic [ 3.435479] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.441535] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.443594] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.446492] mousedev: PS/2 mouse device common for all mice [ 3.450119] rtc_cmos 00:05: RTC can wake from S4 [ 3.452766] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.455282] rtc_cmos 00:05: registered as rtc0 [ 3.459119] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459807] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.464625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.465987] intel_pstate: CPU model not supported [ 3.472089] hid: raw HID events driver (C) Jiri Kosina [ 3.475227] usbcore: registered new interface driver usbhid [ 3.477461] usbhid: USB HID core driver [ 3.479219] drop_monitor: Initializing network drop monitor service [ 3.481795] Initializing XFRM netlink socket [ 3.483862] NET: Registered protocol family 10 [ 3.487601] Segment Routing with IPv6 [ 3.489324] NET: Registered protocol family 17 [ 3.491929] mpls_gso: MPLS GSO support [ 3.498138] RAS: Correctable Errors collector initialized. [ 3.503399] AVX version of gcm_enc/dec engaged. [ 3.505804] AES CTR mode by8 optimization enabled [ 3.585513] sched_clock: Marking stable (3585484806, 0)->(4591415759, -1005930953) [ 3.588810] registered taskstats version 1 [ 3.590991] Loading compiled-in X.509 certificates [ 3.592835] zswap: loaded using pool lzo/zbud [ 3.614957] Key type big_key registered [ 3.627881] Key type encrypted registered [ 3.629755] ima: No TPM chip found, activating TPM-bypass! [ 3.632340] ima: Allocated hash algorithm: sha1 [ 3.634270] ima: No architecture policies found [ 3.636323] evm: Initialising EVM extended attributes: [ 3.638505] evm: security.selinux [ 3.639746] evm: security.ima [ 3.640884] evm: security.capability [ 3.642267] evm: HMAC attrs: 0x1 [ 3.645560] rtc_cmos 00:05: setting system clock to 2026-07-06 14:57:33 UTC (1783349853) [ 3.653510] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.656688] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.665149] Write protecting the kernel read-only data: 28672k [ 3.668376] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.671894] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.709149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.717700] systemd[1]: Detected virtualization kvm. [ 3.720052] systemd[1]: Detected architecture x86-64. [ 3.722234] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.780287] systemd[1]: No hostname configured. [ 3.781560] systemd[1]: Set hostname to . [ 3.784809] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787148] systemd[1]: Initializing machine ID from random generator. [ 3.847380] random: ln: uninitialized urandom read (6 bytes read) [ 3.933456] random: systemd: uninitialized urandom read (16 bytes read) [ 3.936169] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.942423] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.947198] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.648069] device-mapper: uevent: version 1.0.3 [ 4.650754] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.444331] random: fast init done [ 5.447477] virtio_net virtio0 ens2: renamed from eth0 [ 5.535054] scsi host0: ata_piix [ 5.688515] scsi host1: ata_piix [ 5.691576] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.694709] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.311418] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.150128] random: crng init done [ 10.151975] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.706246] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.914153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.203032] SELinux: Disabled at runtime. [ 12.279650] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.290019] systemd[1]: Detected virtualization kvm. [ 12.292248] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.865457] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.869290] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.875382] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.882806] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.886743] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.895392] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.899550] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ 13.072365] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.419792] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.730827] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.778171] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.888197] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.902499] EDAC sbridge: Ver: 1.1.2 [ 15.668834] Key type dns_resolver registered [ 16.184259] NFS: Registering the id_resolver key type [ 16.187358] Key type id_resolver registered [ 16.188714] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg455-server login: [ 51.455682] libcfs: loading out-of-tree module taints kernel. [ 51.477620] Key type ._llcrypt registered [ 51.479102] Key type .llcrypt registered [ 51.522816] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 74.668941] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 77.088380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.140401] alg: No test for adler32 (adler32-zlib) [ 79.053490] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 79.945631] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 81.848242] Key type lgssc registered [ 84.400045] hrtimer: interrupt took 4073437 ns [ 84.578485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.980058] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.993764] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 189.649337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.678317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.066615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.133919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.333689] Lustre: lustre-MDT0000: new disk, initializing [ 191.572915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.611111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.682953] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.035655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.182714] Lustre: 6531:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.212578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.216347] Lustre: Skipped 1 previous similar message [ 214.283362] Lustre: lustre-MDT0001: new disk, initializing [ 214.384038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.410141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.432976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.653553] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.042510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.216032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.537576] Lustre: lustre-OST0000: new disk, initializing [ 236.549731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.570554] Lustre: 8483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.708764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.219209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.225405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.275920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.471497] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.537754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.772799] Lustre: lustre-OST0001: new disk, initializing [ 260.781221] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.791977] Lustre: 9557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.885551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.066500] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.888936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.895567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.977424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.480774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.876897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.387628] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.850763] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing yml_node [ 311.435753] Lustre: DEBUG MARKER: Client: 2.17.54.84 [ 313.949675] Lustre: DEBUG MARKER: MDS: 2.17.54.84 [ 316.207995] Lustre: DEBUG MARKER: OSS: 2.17.54.84 [ 318.208394] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 6 11:02:45 EDT 2026 [ 338.489269] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 340.292754] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.857901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.859573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.878130] Lustre: Skipped 3 previous similar messages [ 357.900701] Lustre: Skipped 3 previous similar messages [ 363.060386] Lustre: server umount lustre-MDT0000 complete [ 366.742260] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783350216 with bad export cookie 3844513323483386197 [ 366.746183] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.749473] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 368.097938] LustreError: 10183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.104822] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.114389] LustreError: 10183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 368.114556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.143934] Lustre: Skipped 1 previous similar message [ 371.106383] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.107714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.127080] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 371.151734] Lustre: Skipped 2 previous similar messages [ 376.231583] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.232032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.247377] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.261833] Lustre: Skipped 1 previous similar message [ 381.421220] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 381.432095] LustreError: 10183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.432283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 381.473342] LustreError: 10183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 381.751355] Lustre: server umount lustre-MDT0001 complete [ 385.746278] Lustre: server umount lustre-OST0000 complete [ 389.878635] Lustre: server umount lustre-OST0001 complete [ 397.289660] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 408.008838] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 448.839489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.188646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.270585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.412334] Lustre: lustre-MDT0000: new disk, initializing [ 449.580164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.606583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.236758] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 466.573172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.765222] Lustre: 15115:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.884316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.888975] Lustre: Skipped 1 previous similar message [ 466.973692] Lustre: lustre-MDT0001: new disk, initializing [ 467.156871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 467.243805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 467.252423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 472.595806] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 483.379050] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 489.814733] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 497.455506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.845952] Lustre: lustre-OST0000: new disk, initializing [ 497.850275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.858693] Lustre: 17086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.985639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.278162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 499.299817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 499.541869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.378520] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 513.368939] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 517.948950] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 518.169167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.716916] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.957906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 525.289755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.292436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.294640] LustreError: Skipped 1 previous similar message [ 525.329142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.336397] Lustre: Skipped 1 previous similar message [ 539.105878] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 539.306280] Lustre: server umount lustre-OST0000 complete [ 544.226088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 544.241891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.248452] Lustre: Skipped 1 previous similar message [ 544.253706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.258146] Lustre: Skipped 5 previous similar messages [ 549.391272] Lustre: server umount lustre-MDT0000 complete [ 550.886161] LustreError: 15122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.910995] LustreError: 15122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 552.868891] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783350402 with bad export cookie 3844513323483387730 [ 552.873883] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.885235] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 553.230747] Lustre: server umount lustre-MDT0001 complete [ 557.180123] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:06:45 (1783350405) [ 559.596196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 561.050845] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:06:49 (1783350409) [ 567.189165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.614252] LustreError: 19497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.718907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.022825] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 581.243482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.410937] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 594.442802] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.713673] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 607.124859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 613.159332] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 620.309722] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 637.409783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.423339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.436694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.436812] Lustre: Skipped 2 previous similar messages [ 637.453611] Lustre: Skipped 2 previous similar messages [ 642.147303] Lustre: server umount lustre-OST0000 complete [ 647.651899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.668882] Lustre: Skipped 1 previous similar message [ 651.712540] Lustre: server umount lustre-MDT0000 complete [ 652.772405] LustreError: 19498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.786797] LustreError: 19498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 654.876409] LustreError: 19478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783350504 with bad export cookie 3844513323483388612 [ 654.883784] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.889830] LustreError: 19478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 655.110692] Lustre: server umount lustre-MDT0001 complete [ 664.883255] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 667.327506] Key type lgssc unregistered [ 667.619972] LNet: 23052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.638194] LNetError: 23052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.665264] LNet: Removed LNI 192.168.204.155@tcp [ 668.507138] Key type .llcrypt unregistered [ 668.509707] Key type ._llcrypt unregistered [ 686.229457] Key type ._llcrypt registered [ 686.235213] Key type .llcrypt registered [ 686.320590] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 686.938863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.015643] alg: No test for adler32 (adler32-zlib) [ 688.047030] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 688.231314] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 689.872949] Key type lgssc registered [ 690.837282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.874027] loop: module loaded [ 697.934665] loop0: detected capacity change from 0 to 204800000 [ 698.156926] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.166597] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.391211] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.439653] loop0: detected capacity change from 0 to 204800000 [ 703.652534] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.665977] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.680462] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.015150] loop0: detected capacity change from 0 to 204800000 [ 715.477747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.518140] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.853829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 716.930020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.001940] Lustre: lustre-MDT0000: new disk, initializing [ 717.129442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.159294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.373653] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 729.439680] loop1: detected capacity change from 0 to 204800000 [ 734.966524] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.332627] Lustre: lustre-OST07c6: new disk, initializing [ 735.336662] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 735.340478] Lustre: Skipped 1 previous similar message [ 735.346824] Lustre: 26217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 735.424533] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 741.214743] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 741.465630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 741.483196] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 741.535677] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 745.452947] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 745.469668] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 745.478890] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 745.491505] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 745.505390] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 745.522044] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 764.116274] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 764.127593] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.135620] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.143137] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 764.150611] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 764.165283] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.113256] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 769.125654] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.136028] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 769.147470] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.154155] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.160781] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.166807] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 769.173047] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.176021] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.179334] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.184307] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.187837] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.344208] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 772.352872] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.360191] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.367515] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.374209] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.379619] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.391133] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 772.402815] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.411295] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.418625] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.425048] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.432993] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.674531] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 776.686537] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.695795] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.702565] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.710151] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.715210] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.720699] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 776.727898] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.733289] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.738334] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.745089] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.751151] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.533193] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 787.545383] Lustre: 26899:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 787.554336] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.563633] Lustre: 26899:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.569368] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.574626] Lustre: 26899:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.581111] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 787.587139] Lustre: 26899:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.593188] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 787.598707] Lustre: 26899:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 787.604853] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 787.611213] Lustre: 26899:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 838.627555] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.639946] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.656641] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 839.882765] Lustre: server umount lustre-OST07c6 complete [ 848.228234] Lustre: server umount lustre-MDT0000 complete [ 863.401304] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 865.554637] Key type lgssc unregistered [ 865.830679] LNet: 28773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.842795] LNetError: 28773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.858687] LNet: Removed LNI 192.168.204.155@tcp [ 866.570752] Key type .llcrypt unregistered [ 866.573416] Key type ._llcrypt unregistered [ 881.778517] Key type ._llcrypt registered [ 881.782129] Key type .llcrypt registered [ 881.916086] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 882.669472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.730346] alg: No test for adler32 (adler32-zlib) [ 883.730993] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 883.934397] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 885.561516] Key type lgssc registered [ 886.517946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.841732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.860248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.185548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.665443] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 904.537675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.680199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 904.895055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.977051] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 912.839751] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.400594] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 920.973255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.236853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.459498] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 930.099682] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 939.705051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 955.368915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.369544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.389180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.013588] Lustre: server umount lustre-OST0000 complete [ 960.483979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.485977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.498534] Lustre: Skipped 2 previous similar messages [ 960.508546] Lustre: Skipped 2 previous similar messages [ 965.602897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.608559] Lustre: Skipped 1 previous similar message [ 965.901615] Lustre: server umount lustre-MDT0000 complete [ 968.242416] LustreError: 32737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783350818 with bad export cookie 762494803867861712 [ 968.246815] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.253055] LustreError: 32737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 968.445690] Lustre: server umount lustre-MDT0001 complete [ 975.838820] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 977.946627] Key type lgssc unregistered [ 978.186826] LNet: 33781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.195719] LNetError: 33781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.206458] LNet: Removed LNI 192.168.204.155@tcp [ 978.942975] Key type .llcrypt unregistered [ 978.945445] Key type ._llcrypt unregistered [ 992.480865] Key type ._llcrypt registered [ 992.485417] Key type .llcrypt registered [ 992.575747] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 993.168800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 993.206438] alg: No test for adler32 (adler32-zlib) [ 994.158580] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 994.323303] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 995.976252] Key type lgssc registered [ 996.763820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.307875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.326078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.637905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.866859] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1013.218760] LustreError: 35028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.235941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.241036] LustreError: 35028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1013.541369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1016.685380] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1021.930356] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.220350] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1030.793251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.032704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.703904] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1039.542692] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.430926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1058.786365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.789738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.792615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.794790] Lustre: Skipped 1 previous similar message [ 1062.961896] Lustre: server umount lustre-OST0000 complete [ 1069.029281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.037485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.042254] Lustre: Skipped 1 previous similar message [ 1069.049979] Lustre: Skipped 2 previous similar messages [ 1071.297432] Lustre: server umount lustre-MDT0000 complete [ 1073.689853] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783350923 with bad export cookie 14325269222446319245 [ 1073.693277] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1073.697640] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1073.916424] Lustre: server umount lustre-MDT0001 complete [ 1080.816966] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1082.653962] Key type lgssc unregistered [ 1082.850509] LNet: 38582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.858879] LNetError: 38582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.880854] LNet: Removed LNI 192.168.204.155@tcp [ 1084.436942] Key type .llcrypt unregistered [ 1084.439107] Key type ._llcrypt unregistered [ 1101.354559] Key type ._llcrypt registered [ 1101.356061] Key type .llcrypt registered [ 1101.433215] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1102.069404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.091247] alg: No test for adler32 (adler32-zlib) [ 1103.071514] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1103.224514] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1104.865264] Key type lgssc registered [ 1105.614921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.749924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.764314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.042340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.682302] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1120.726728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.967490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.764257] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1129.242556] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1133.310533] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1138.018909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1138.249920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.501408] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1147.882576] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1156.729059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1164.643036] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 1167.333365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.333727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.336301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.341317] Lustre: Skipped 1 previous similar message [ 1168.562101] Lustre: server umount lustre-MDT0000 complete [ 1169.732283] LustreError: 40784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.743676] LustreError: 40784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1170.830459] LustreError: 40710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351020 with bad export cookie 18186872108611103305 [ 1170.837304] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.838297] LustreError: 40710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1172.449820] LustreError: 39971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.452413] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.474750] Lustre: Skipped 1 previous similar message [ 1172.477777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1172.480417] Lustre: Skipped 2 previous similar messages [ 1174.856613] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.873133] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 1179.970357] LustreError: 39971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.971196] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 1179.987456] LustreError: 39971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1180.002026] Lustre: Skipped 1 previous similar message [ 1185.092478] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.107481] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1185.248134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1185.415332] Lustre: server umount lustre-MDT0001 complete [ 1198.989617] Lustre: server umount lustre-OST0000 complete [ 1211.535982] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1213.521773] Key type lgssc unregistered [ 1213.722039] LNet: 43853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.732184] LNetError: 43853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.743183] LNet: Removed LNI 192.168.204.155@tcp [ 1214.292530] Key type .llcrypt unregistered [ 1214.294540] Key type ._llcrypt unregistered [ 1232.691465] Key type ._llcrypt registered [ 1232.696893] Key type .llcrypt registered [ 1232.793901] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1233.667799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.712353] alg: No test for adler32 (adler32-zlib) [ 1234.719988] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1234.961623] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1236.616826] Key type lgssc registered [ 1237.499861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.695843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.713343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.068752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.866112] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1252.781208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.006489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.943075] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1260.968817] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.848408] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1269.634916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.898945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.140480] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1279.607399] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.363685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1296.722685] Lustre: lustre-OST0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 1298.916498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.918663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.924917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.929294] LustreError: Skipped 1 previous similar message [ 1298.937198] Lustre: Skipped 1 previous similar message [ 1301.434908] Lustre: server umount lustre-OST0000 complete [ 1317.479649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.488598] Lustre: Skipped 1 previous similar message [ 1317.492045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.495691] Lustre: Skipped 1 previous similar message [ 1323.792259] Lustre: server umount lustre-MDT0000 complete [ 1324.001335] LustreError: 46592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.015064] LustreError: 46592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1326.294525] LustreError: 45230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351176 with bad export cookie 11191007490987262754 [ 1326.300465] LustreError: 45230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1326.301263] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.550573] Lustre: server umount lustre-MDT0001 complete [ 1338.713261] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1340.332200] Key type lgssc unregistered [ 1340.521677] LNet: 48980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.530391] LNetError: 48980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.543109] LNet: Removed LNI 192.168.204.155@tcp [ 1341.054616] Key type .llcrypt unregistered [ 1341.056810] Key type ._llcrypt unregistered [ 1356.355207] Key type ._llcrypt registered [ 1356.358500] Key type .llcrypt registered [ 1356.429828] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1357.012938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.097164] alg: No test for adler32 (adler32-zlib) [ 1358.043102] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1358.202878] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1359.840197] Key type lgssc registered [ 1360.627572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.212485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.240824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.571046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.528428] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1376.004528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.228326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1378.763271] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1384.009891] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1387.958559] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1392.488610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.567061] LustreError: 52196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.585390] LustreError: 52196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1392.796723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.064365] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1398.262729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1402.438323] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1406.501853] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1408.470153] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1419.788809] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-0611h21m07s [ 1420.831181] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1422.306307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.313766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.324949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.396989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.401325] Lustre: Skipped 4 previous similar messages [ 1424.725545] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 1428.426594] Lustre: server umount lustre-MDT0000 complete [ 1429.477513] LustreError: 51187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.493304] LustreError: 51187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1430.438589] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1430.858236] LustreError: 53060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.668028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.745457] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.898419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.943733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1435.947852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1437.628470] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1439.642495] Lustre: DEBUG MARKER: Wait for df (36603) ... [ 1440.613452] Lustre: DEBUG MARKER: done [ 1445.346868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.351264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.365925] Lustre: Skipped 2 previous similar messages [ 1445.374478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.379297] Lustre: Skipped 1 previous similar message [ 1446.370158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.390738] Lustre: server umount lustre-OST0000 complete [ 1456.611517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.614074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.624632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.634880] Lustre: Skipped 1 previous similar message [ 1456.639752] Lustre: Skipped 2 previous similar messages [ 1457.699844] LustreError: 50376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.711968] LustreError: 50376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1457.774299] Lustre: server umount lustre-MDT0000 complete [ 1460.193528] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351310 with bad export cookie 18239615358823419350 [ 1460.196647] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1460.203787] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1460.405767] Lustre: server umount lustre-MDT0001 complete [ 1467.094599] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1468.769837] Key type lgssc unregistered [ 1468.956792] LNet: 55297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.965878] LNetError: 55297:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.975960] LNet: Removed LNI 192.168.204.155@tcp [ 1469.467547] Key type .llcrypt unregistered [ 1469.469307] Key type ._llcrypt unregistered [ 1496.067586] Key type ._llcrypt registered [ 1496.070579] Key type .llcrypt registered [ 1496.137176] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1496.846913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.884435] alg: No test for adler32 (adler32-zlib) [ 1497.825429] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1498.014728] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1499.649220] Key type lgssc registered [ 1500.297813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.247322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1505.268928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.435156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1506.451731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1506.590185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.303275] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1515.516374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.598930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1515.624838] Lustre: 57484:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1515.809443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1518.823703] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1523.767808] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1527.818101] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1532.056489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.189271] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1532.302040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.281651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1536.616465] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1540.968230] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1546.230746] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-0611h23m14s [ 1547.065903] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1550.722602] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1551.956856] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1551.989965] LustreError: 57493:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1572.200192] Lustre: lustre-MDT0000: Client b96e64a4-c158-4f0a-8361-37f32d9b3ee7 (at 192.168.204.55@tcp) reconnecting [ 1592.056091] LustreError: 57493:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1593.661475] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1595.619859] Lustre: DEBUG MARKER: done [ 1599.970459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.975658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1599.995980] Lustre: Skipped 1 previous similar message [ 1599.996560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.014390] Lustre: Skipped 1 previous similar message [ 1605.093517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1605.096716] Lustre: Skipped 1 previous similar message [ 1605.525849] Lustre: server umount lustre-OST0000 complete [ 1608.165018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.174919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.192254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.106627] Lustre: server umount lustre-MDT0000 complete [ 1616.699646] LustreError: 58226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351466 with bad export cookie 12500302487936051943 [ 1616.700494] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.721103] LustreError: 58226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1616.932767] Lustre: server umount lustre-MDT0001 complete [ 1623.579615] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1625.519161] Key type lgssc unregistered [ 1625.748169] LNet: 61809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.761846] LNetError: 61809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.773550] LNet: Removed LNI 192.168.204.155@tcp [ 1626.333171] Key type .llcrypt unregistered [ 1626.335050] Key type ._llcrypt unregistered [ 1655.181391] Key type ._llcrypt registered [ 1655.184025] Key type .llcrypt registered [ 1655.247963] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1655.979936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1656.007443] alg: No test for adler32 (adler32-zlib) [ 1656.967523] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1657.170868] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1658.808237] Key type lgssc registered [ 1659.639372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.948826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.960973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.167766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1663.189977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1663.314870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.093685] Lustre: Failing over lustre-MDT0000 [ 1664.264828] Lustre: server umount lustre-MDT0000 complete [ 1668.934236] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:25:17 (1783351517) [ 1673.392080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.680604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.121568] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1681.467789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.537850] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1681.564953] Lustre: 64589:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1681.749686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1684.214779] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1688.017482] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1691.052541] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1694.943479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.061818] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1695.180735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.381841] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1702.208694] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1703.993407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1712.423535] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1717.730196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1717.737063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.745208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1718.767805] Lustre: server umount lustre-MDT0000 complete [ 1719.269043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.909961] LustreError: 65323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351570 with bad export cookie 4132227224225184907 [ 1720.910539] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.919870] LustreError: 65323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1721.089977] Lustre: server umount lustre-MDT0001 complete [ 1722.925952] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1725.516794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1730.243561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.528661] LustreError: 63509:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007e0bb4a0 x1869979589839872/t0(0) o250->MGC192.168.204.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1730.702232] LustreError: 66435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.715028] LustreError: 66435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1730.745348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.794732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1732.807844] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1736.177638] LustreError: 66436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.181259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.198635] Lustre: Skipped 1 previous similar message [ 1737.616071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.944312] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1740.795479] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1740.961109] Lustre: 63512:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783351574/real 1783351574] req@00000000a43fc701 x1869979589839488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783351590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1742.305318] Lustre: 63511:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783351575/real 1783351575] req@000000001ef38e0a x1869979589839616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783351591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1745.177860] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.376310] Lustre: 63511:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783351579/real 1783351579] req@000000003f9c7b1f x1869979589839744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783351595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1748.601689] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1753.571902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.575499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.585053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.623511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.634379] Lustre: Skipped 2 previous similar messages [ 1758.678402] Lustre: server umount lustre-MDT0000 complete [ 1760.740775] LustreError: 69889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.758772] LustreError: 69889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1761.165441] LustreError: 66442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351611 with bad export cookie 4132227224225187168 [ 1761.169776] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.175583] LustreError: 66442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1761.326239] Lustre: server umount lustre-MDT0001 complete [ 1765.106218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1770.909012] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.002607] LustreError: 71019:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1772.006860] LustreError: 63509:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ffec204c x1869979589863040/t0(0) o250->MGC192.168.204.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1772.032791] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x39589ec457240ac3 [ 1772.044654] Lustre: MGC192.168.204.155@tcp: Connection restored to 0@lo (at 0@lo) [ 1772.051879] Lustre: Skipped 1 previous similar message [ 1772.210790] LustreError: 66435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.271481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.279062] Lustre: Skipped 1 previous similar message [ 1772.316907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1775.058669] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1777.639156] LustreError: 66436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.642342] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.666678] Lustre: Skipped 2 previous similar messages [ 1780.737768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.048707] Lustre: 63511:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783351615/real 1783351615] req@00000000431dacb3 x1869979589862656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783351631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1782.083056] Lustre: 63511:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1784.307483] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1786.089837] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1787.106171] Lustre: 63513:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783351620/real 1783351620] req@00000000f8b7954e x1869979589862912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783351636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1787.125020] Lustre: 63513:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1789.273462] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1793.188677] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1801.701628] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.702245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1801.704253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.710192] Lustre: Skipped 1 previous similar message [ 1803.525610] Lustre: server umount lustre-MDT0000 complete [ 1805.985876] LustreError: 64579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351655 with bad export cookie 4132227224225188547 [ 1805.990662] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.997673] LustreError: 64579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1806.198969] Lustre: server umount lustre-MDT0001 complete [ 1810.791906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1813.198876] Lustre: DEBUG MARKER: files compared the same [ 1824.766568] Lustre: server umount lustre-OST0000 complete [ 1836.665442] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1838.489477] Key type lgssc unregistered [ 1838.695701] LNet: 74303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.706335] LNetError: 74303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.727727] LNet: Removed LNI 192.168.204.155@tcp [ 1839.323676] Key type .llcrypt unregistered [ 1839.326121] Key type ._llcrypt unregistered [ 1855.884906] Key type ._llcrypt registered [ 1855.887467] Key type .llcrypt registered [ 1855.985399] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1856.539538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.553932] alg: No test for adler32 (adler32-zlib) [ 1857.535064] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1857.755747] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1859.392181] Key type lgssc registered [ 1860.142698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1865.154345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1865.179885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.502469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.044450] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug +malloc all [ 1874.869303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.044153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.160850] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug +malloc all [ 1882.500622] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1885.770332] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.896682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1890.072567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.136437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1893.559238] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug +malloc all [ 1897.697955] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1904.610141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.611881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.621488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.634142] Lustre: Skipped 1 previous similar message [ 1904.650690] Lustre: Skipped 1 previous similar message [ 1906.368424] Lustre: server umount lustre-OST0000 complete [ 1909.729432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.730157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.738490] Lustre: Skipped 1 previous similar message [ 1909.744932] Lustre: Skipped 1 previous similar message [ 1914.849041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.854878] Lustre: Skipped 1 previous similar message [ 1914.991715] Lustre: server umount lustre-MDT0000 complete [ 1917.369322] LustreError: 75678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351767 with bad export cookie 1575425083080271653 [ 1917.377940] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.379853] LustreError: 75678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1917.596243] Lustre: server umount lustre-MDT0001 complete [ 1925.409840] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1927.334262] Key type lgssc unregistered [ 1927.533640] LNet: 79228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.539429] LNetError: 79228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.551732] LNet: Removed LNI 192.168.204.155@tcp [ 1928.099720] Key type .llcrypt unregistered [ 1928.102384] Key type ._llcrypt unregistered [ 1944.817812] Key type ._llcrypt registered [ 1944.819657] Key type .llcrypt registered [ 1944.893814] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1945.323490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.330760] alg: No test for adler32 (adler32-zlib) [ 1946.256467] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 1946.389800] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1948.008132] Key type lgssc registered [ 1948.670974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.205596] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1953.216946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1968.672566] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1973.792581] LustreError: 80606:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 2000.416656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.194322] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2007.146273] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2011.036040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.104224] Lustre: *** cfs_fail_loc=706, val=0*** [ 2011.317724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.355671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2013.724466] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2018.378716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.576382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.724507] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2024.386852] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2027.202829] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2029.026019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2029.028908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.030606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2029.035972] Lustre: Skipped 1 previous similar message [ 2034.148054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.151695] Lustre: Skipped 2 previous similar messages [ 2034.346247] Lustre: server umount lustre-OST0000 complete [ 2039.265538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.266751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2039.267800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.272236] Lustre: Skipped 1 previous similar message [ 2042.752654] Lustre: server umount lustre-MDT0000 complete [ 2044.387207] LustreError: 81690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.397860] LustreError: 81690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2044.684806] LustreError: 80614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783351894 with bad export cookie 15580508331302015930 [ 2044.687982] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.692631] LustreError: 80614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.826236] Lustre: server umount lustre-MDT0001 complete [ 2050.039360] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2051.585854] Key type lgssc unregistered [ 2051.747889] LNet: 84217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.754468] LNetError: 84217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.771277] LNet: Removed LNI 192.168.204.155@tcp [ 2052.245210] Key type .llcrypt unregistered [ 2052.248074] Key type ._llcrypt unregistered [ 2065.460846] Key type ._llcrypt registered [ 2065.462516] Key type .llcrypt registered [ 2065.519438] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2066.081867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.107712] alg: No test for adler32 (adler32-zlib) [ 2067.014402] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2067.155351] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2068.760551] Key type lgssc registered [ 2069.301855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2072.948035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2072.961953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.985298] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2080.128159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.430698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.677493] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2085.863736] LustreError: 86300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.873302] LustreError: 86300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2087.201943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.379990] LustreError: 86299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.388807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.390037] LustreError: 86299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2090.469525] LustreError: 86304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.486820] LustreError: 86304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2090.514870] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2095.467841] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.482875] Lustre: MGS: Client 5f5c7294-a612-4971-a75a-c15af26bb3b9 (at 0@lo) reconnecting [ 2095.499089] LustreError: 86314:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2095.630204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.666540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2098.053035] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2110.945984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.948808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.950197] LustreError: Skipped 1 previous similar message [ 2110.957589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.963491] Lustre: Skipped 1 previous similar message [ 2114.719345] Lustre: server umount lustre-OST0000 complete [ 2121.186687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2121.188599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.193731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.200455] Lustre: Skipped 2 previous similar messages [ 2122.585469] Lustre: server umount lustre-MDT0000 complete [ 2124.391741] Lustre: server umount lustre-MDT0001 complete [ 2126.029261] Lustre: server umount MGS complete [ 2130.903328] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:32:59 (1783351979) [ 2144.356612] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 2148.678859] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2165.710704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.121907] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2174.040514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.167816] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2174.193145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2174.246420] Lustre: lustre-MDT0001: new disk, initializing [ 2174.294486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.309794] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2174.314159] Lustre: 93458:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2174.321043] LustreError: 93458:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2176.695255] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2182.712677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.845081] Lustre: lustre-OST0000: new disk, initializing [ 2182.847813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2182.853802] Lustre: 94225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2182.885094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.609196] Lustre: 94306:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ef0b08dd x1869980019756544/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783352050 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2185.767333] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2190.612242] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2194.115867] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.124641] Lustre: MGS: Client 7123b07c-6612-481e-987d-b5023a91245c (at 0@lo) reconnecting [ 2194.146403] Lustre: 92726:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2194.158706] LustreError: 93444:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2194.180170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2194.227217] Lustre: lustre-MDT0000: new disk, initializing [ 2194.295776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.310164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2194.357530] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2195.360788] Lustre: 95294:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006071fee9 x1869980019762944/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783352061 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2196.135133] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2204.461145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2204.468374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2204.544958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2228.196266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.198463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.203723] Lustre: Skipped 1 previous similar message [ 2228.206528] Lustre: Skipped 1 previous similar message [ 2232.756928] Lustre: server umount lustre-OST0000 complete [ 2238.433582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2238.434654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.437834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.451129] Lustre: Skipped 1 previous similar message [ 2238.456607] Lustre: Skipped 2 previous similar messages [ 2240.853452] Lustre: server umount lustre-MDT0000 complete [ 2257.376133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2257.490152] Lustre: server umount lustre-MDT0001 complete [ 2259.654028] Lustre: server umount MGS complete [ 2265.852124] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:35:14 (1783352114) [ 2274.849393] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2276.475945] Key type lgssc unregistered [ 2276.658523] LNet: 97792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.662690] LNetError: 97792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.736881] LNet: Removed LNI 192.168.204.155@tcp [ 2278.190072] Key type .llcrypt unregistered [ 2278.191692] Key type ._llcrypt unregistered [ 2287.512202] Key type ._llcrypt registered [ 2287.514114] Key type .llcrypt registered [ 2287.579723] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2287.980199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.989320] alg: No test for adler32 (adler32-zlib) [ 2288.918691] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2289.048587] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2290.648207] Key type lgssc registered [ 2291.187138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.084270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.093550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.325118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.199095] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2301.893225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.080342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2303.961664] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2307.497652] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2310.222848] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2313.466969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2313.634791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.424973] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2319.846745] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2321.837111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2332.129579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2332.130402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.136794] LustreError: Skipped 1 previous similar message [ 2332.144818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.265796] Lustre: server umount lustre-OST0000 complete [ 2342.370238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.371170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2342.376310] Lustre: Skipped 2 previous similar messages [ 2342.380244] Lustre: Skipped 1 previous similar message [ 2345.405394] Lustre: server umount lustre-MDT0000 complete [ 2347.264650] LustreError: 98975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352197 with bad export cookie 12871405196302314073 [ 2347.265544] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.272133] LustreError: 98975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2347.435439] Lustre: server umount lustre-MDT0001 complete [ 2353.932471] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2355.299172] Key type lgssc unregistered [ 2355.472763] LNet: 102541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.478063] LNetError: 102541:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.489517] LNet: Removed LNI 192.168.204.155@tcp [ 2355.918920] Key type .llcrypt unregistered [ 2355.922765] Key type ._llcrypt unregistered [ 2364.456986] Key type ._llcrypt registered [ 2364.458384] Key type .llcrypt registered [ 2364.514479] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2364.999886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.011863] alg: No test for adler32 (adler32-zlib) [ 2365.903693] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2366.005326] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2367.592156] Key type lgssc registered [ 2368.229340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.122533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2371.122603] LustreError: 103601:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2371.126022] Lustre: Skipped 1 previous similar message [ 2371.132341] LustreError: 103602:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2371.135613] LustreError: 103601:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2371.139492] LustreError: 103601:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2371.144741] LustreError: 103601:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2371.147810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.368754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.919664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.055993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.969279] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2380.255961] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing lsmod [ 2381.563685] LustreError: 104807:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2381.563684] LustreError: 104806:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2381.571240] LustreError: 104807:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2381.575618] LustreError: 104807:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2381.581454] LustreError: 104807:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2381.584285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.756791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.769839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2385.189929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.194092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.197642] Lustre: Skipped 2 previous similar messages [ 2385.206400] Lustre: Skipped 2 previous similar messages [ 2389.304213] Lustre: server umount lustre-MDT0000 complete [ 2390.499426] LustreError: 103641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.507361] LustreError: 103641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2390.933240] LustreError: 103620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352240 with bad export cookie 12075124198175736083 [ 2390.934656] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.937473] LustreError: 103620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2390.942545] Lustre: 103632:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2391.352284] LustreError: 104070:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000006f5e843 x1869980332213632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2391.436951] Lustre: server umount lustre-MDT0001 complete [ 2402.868179] Lustre: server umount lustre-OST0000 complete [ 2407.246701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.452337] LustreError: 105870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.485257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.411724] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2412.514308] LustreError: 105871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.319621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.517226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.438660] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2418.954745] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2421.388293] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2424.552297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.714734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.258422] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2430.655510] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2432.935164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2444.257934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.261041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.263644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.277323] Lustre: Skipped 1 previous similar message [ 2446.176643] Lustre: server umount lustre-OST0000 complete [ 2447.912908] Lustre: server umount lustre-MDT0000 complete [ 2449.377989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.378060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.379147] LustreError: 105865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.396198] Lustre: Skipped 1 previous similar message [ 2449.487206] LustreError: 105852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352299 with bad export cookie 12075124198175736993 [ 2449.491851] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.493426] LustreError: 105852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.703316] Lustre: server umount lustre-MDT0001 complete [ 2460.443878] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2461.877723] Key type lgssc unregistered [ 2462.037560] LNet: 109421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.040557] LNetError: 109421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.055509] LNet: Removed LNI 192.168.204.155@tcp [ 2462.464363] Key type .llcrypt unregistered [ 2462.466140] Key type ._llcrypt unregistered [ 2474.533245] Key type ._llcrypt registered [ 2474.534878] Key type .llcrypt registered [ 2474.587261] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2475.067956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.079534] alg: No test for adler32 (adler32-zlib) [ 2475.984763] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2476.102592] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2477.704219] Key type lgssc registered [ 2478.159469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2481.387623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2481.400751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.632719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.489395] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2488.578863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.743796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.625091] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2494.248549] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2496.696591] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2500.370939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.552073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.327640] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2506.670890] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2507.754118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2519.290041] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2520.760794] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2523.106175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.110616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.117099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.132871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.132971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.150305] Lustre: Skipped 1 previous similar message [ 2528.796956] Lustre: server umount lustre-OST0000 complete [ 2534.376359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.377678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.381551] Lustre: Skipped 1 previous similar message [ 2534.385787] Lustre: Skipped 1 previous similar message [ 2536.607839] Lustre: server umount lustre-MDT0000 complete [ 2538.069219] LustreError: 110796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352387 with bad export cookie 16355014265000286651 [ 2538.073183] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.074116] LustreError: 110796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2538.224179] Lustre: server umount lustre-MDT0001 complete [ 2542.648987] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2543.868201] Key type lgssc unregistered [ 2544.019733] LNet: 114474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.024669] LNetError: 114474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.038517] LNet: Removed LNI 192.168.204.155@tcp [ 2544.374779] Key type .llcrypt unregistered [ 2544.376727] Key type ._llcrypt unregistered [ 2552.508046] Key type ._llcrypt registered [ 2552.509674] Key type .llcrypt registered [ 2552.549189] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2552.954774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2553.017487] alg: No test for adler32 (adler32-zlib) [ 2553.902682] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2554.014961] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2555.616174] Key type lgssc registered [ 2556.062156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2559.214803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2559.223206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.400704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.939861] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2565.229410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.381448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2566.937777] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2569.973958] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2572.132057] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2575.069953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.224107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.773696] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2580.757547] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2585.448324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2590.690852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.694562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.700679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.703346] Lustre: Skipped 1 previous similar message [ 2595.290601] Lustre: server umount lustre-OST0000 complete [ 2600.930497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.931344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.934347] Lustre: Skipped 2 previous similar messages [ 2606.049383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.052936] Lustre: Skipped 2 previous similar messages [ 2611.168169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2611.170145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.176468] Lustre: Skipped 1 previous similar message [ 2611.248831] Lustre: server umount lustre-MDT0000 complete [ 2612.853410] LustreError: 116407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352462 with bad export cookie 16083390516927357075 [ 2612.855594] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.868639] LustreError: 116407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2613.008525] Lustre: server umount lustre-MDT0001 complete [ 2617.383000] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2618.823536] Key type lgssc unregistered [ 2618.984439] LNet: 119205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2618.988882] LNetError: 119205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.002352] LNet: Removed LNI 192.168.204.155@tcp [ 2619.457045] Key type .llcrypt unregistered [ 2619.460551] Key type ._llcrypt unregistered [ 2630.518805] Key type ._llcrypt registered [ 2630.520792] Key type .llcrypt registered [ 2630.575420] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2631.072944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.096748] alg: No test for adler32 (adler32-zlib) [ 2631.991119] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2632.117575] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2633.712260] Key type lgssc registered [ 2634.280542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2637.664925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2637.674238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.915863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.770990] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2644.618906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.794832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.596764] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2649.916529] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2652.159236] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2655.122625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.288449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.571132] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2660.705489] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2663.468470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2674.962394] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2676.562036] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2676.565038] Lustre: Skipped 2 previous similar messages [ 2678.391156] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2678.396876] Lustre: Skipped 2 previous similar messages [ 2680.840266] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2680.854324] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2689.865433] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2689.870071] Lustre: Skipped 2 previous similar messages [ 2698.250448] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.55@tcp 192.168.0.[2,10]@tcp [ 2701.066614] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.55/24@tcp in log lustre-MDT0000 [ 2701.071982] Lustre: Skipped 2 previous similar messages [ 2704.064830] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.55/24@tcp [ 2704.068635] Lustre: Skipped 1 previous similar message [ 2711.010704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2711.014247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.020648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.791913] Lustre: server umount lustre-OST0000 complete [ 2715.617540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.618381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.623237] Lustre: Skipped 1 previous similar message [ 2720.349575] Lustre: server umount lustre-MDT0000 complete [ 2720.737267] LustreError: 120598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.744982] LustreError: 120598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2721.707121] LustreError: 120577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352571 with bad export cookie 4573414516442181708 [ 2721.709752] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.712454] LustreError: 120577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2732.063850] Lustre: server umount lustre-MDT0001 complete [ 2736.275046] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2737.447035] Key type lgssc unregistered [ 2737.605112] LNet: 125696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.609853] LNetError: 125696:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2737.619753] LNet: Removed LNI 192.168.204.155@tcp [ 2737.991919] Key type .llcrypt unregistered [ 2737.993982] Key type ._llcrypt unregistered [ 2749.496783] Key type ._llcrypt registered [ 2749.498588] Key type .llcrypt registered [ 2749.541709] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2749.984885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.004611] alg: No test for adler32 (adler32-zlib) [ 2750.869311] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2750.972606] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2752.560166] Key type lgssc registered [ 2753.022351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2756.124774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2756.133567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.322854] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2757.329134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.335396] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.55/24@tcp [ 2758.917548] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2762.204742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.363759] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2762.373508] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.55/24@tcp [ 2762.401647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2764.058563] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2767.170647] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2769.466129] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2772.405795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.574465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.041870] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2778.069821] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2782.824784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2788.323416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.324584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.330467] Lustre: Skipped 1 previous similar message [ 2788.335731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.727225] Lustre: server umount lustre-OST0000 complete [ 2793.375638] Lustre: server umount lustre-MDT0000 complete [ 2793.442343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.443526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.447172] LustreError: 127231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.451029] Lustre: Skipped 1 previous similar message [ 2793.460257] LustreError: 127231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2794.990204] LustreError: 127212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352644 with bad export cookie 12510703209608760905 [ 2794.994051] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.994531] LustreError: 127212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2795.131948] Lustre: server umount lustre-MDT0001 complete [ 2799.471172] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2800.654314] Key type lgssc unregistered [ 2800.803575] LNet: 130815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2800.812507] LNetError: 130815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2800.820220] LNet: Removed LNI 192.168.204.155@tcp [ 2801.208389] Key type .llcrypt unregistered [ 2801.209578] Key type ._llcrypt unregistered [ 2811.707795] Key type ._llcrypt registered [ 2811.709116] Key type .llcrypt registered [ 2811.755464] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 2818.923828] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2819.241781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2819.269367] alg: No test for adler32 (adler32-zlib) [ 2820.148101] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2820.256554] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2821.848852] Key type lgssc registered [ 2822.304254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2836.327078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2836.334244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.431131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2837.443601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2837.484398] Lustre: lustre-MDT0000: new disk, initializing [ 2837.507683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.516634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2839.009372] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2843.373512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.417275] Lustre: 134189:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2843.442078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2843.444413] Lustre: Skipped 1 previous similar message [ 2843.480762] Lustre: lustre-MDT0001: new disk, initializing [ 2843.516903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.531751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2843.535932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2844.991698] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2849.031411] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.135722] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2853.964313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.089673] Lustre: lustre-OST0000: new disk, initializing [ 2854.092470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2854.095674] Lustre: 136160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2854.130149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.875081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2855.880134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2855.921276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2856.488599] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2860.897362] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2862.663829] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2862.760662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2864.432761] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2864.521139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2866.148540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2866.150360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.151328] LustreError: Skipped 1 previous similar message [ 2866.151432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2866.156056] Lustre: Skipped 1 previous similar message [ 2871.266224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.268605] Lustre: Skipped 2 previous similar messages [ 2871.314800] Lustre: server umount lustre-OST0000 complete [ 2874.336828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2874.339604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.344504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.386164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.386513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.398062] Lustre: Skipped 1 previous similar message [ 2878.818551] Lustre: server umount lustre-MDT0000 complete [ 2880.148647] LustreError: 134183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352730 with bad export cookie 12243011492041949703 [ 2880.150712] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.156307] LustreError: 134183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2880.275312] Lustre: server umount lustre-MDT0001 complete [ 2883.594798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.880708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.486793] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2888.640885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.798341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.351133] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2893.006255] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2895.024387] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2897.465490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2897.574311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.534313] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2901.997098] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2903.403541] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2909.606553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2911.046828] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2911.121919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2914.704463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.769725] Lustre: lustre-OST0001: new disk, initializing [ 2914.771457] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2914.774769] Lustre: 141779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2914.803852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2916.931752] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2920.474318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2920.480246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2920.515797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2920.679573] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2920.767340] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2922.468627] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2922.572769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2924.974413] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d0587f2d000.ost_server_uuid 50 [ 2925.603948] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d0587f2d000.ost_server_uuid in FULL state after 0 sec [ 2930.660471] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2930.661110] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.670198] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2930.672600] Lustre: Skipped 1 previous similar message [ 2933.905858] Lustre: server umount lustre-OST0001 complete [ 2935.778686] LustreError: 140373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.779174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.785099] LustreError: 140373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2935.790739] Lustre: Skipped 1 previous similar message [ 2940.897562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.899936] LustreError: 140189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.900559] Lustre: Skipped 3 previous similar messages [ 2941.665676] Lustre: server umount lustre-OST0000 complete [ 2944.992931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2944.997024] LustreError: Skipped 1 previous similar message [ 2944.999640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.006071] Lustre: Skipped 1 previous similar message [ 2949.414261] Lustre: server umount lustre-MDT0000 complete [ 2950.930537] LustreError: 138351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352800 with bad export cookie 12243011492041950571 [ 2950.932849] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2950.935801] LustreError: 138351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2951.067252] Lustre: server umount lustre-MDT0001 complete [ 2958.966779] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2960.146876] Key type lgssc unregistered [ 2960.280268] LNet: 144780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.284166] LNetError: 144780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.298476] LNet: Removed LNI 192.168.204.155@tcp [ 2960.649061] Key type .llcrypt unregistered [ 2960.650582] Key type ._llcrypt unregistered [ 2977.679457] Key type ._llcrypt registered [ 2977.680442] Key type .llcrypt registered [ 2977.720322] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 2984.542693] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2984.996322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2985.028358] alg: No test for adler32 (adler32-zlib) [ 2985.907061] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 2986.017379] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2987.608212] Key type lgssc registered [ 2988.031146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3001.872786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.882447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.974683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3002.988987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3003.030763] Lustre: lustre-MDT0000: new disk, initializing [ 3003.057074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.064878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3004.683378] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3009.372105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.423057] Lustre: 148905:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3009.452236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3009.456224] Lustre: Skipped 1 previous similar message [ 3009.497486] Lustre: lustre-MDT0001: new disk, initializing [ 3009.544699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.565927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3009.570634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3011.171631] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3015.139592] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3017.094506] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3019.748934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.876484] Lustre: lustre-OST0000: new disk, initializing [ 3019.878723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3019.885145] Lustre: 150872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.925667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.057450] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3025.915733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3025.920230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3025.964903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3026.113904] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.351042] Lustre: Failing over lustre-OST0000 [ 3028.415973] Lustre: server umount lustre-OST0000 complete [ 3041.744835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.903273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.913816] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3042.640587] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3044.206740] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3045.738656] Lustre: Failing over lustre-MDT0000 [ 3045.841388] Lustre: server umount lustre-MDT0000 complete [ 3046.752181] Lustre: 147136:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783352880/real 1783352880] req@00000000d36749f8 x1869980982339840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783352896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3046.752467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.765852] Lustre: 147136:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3051.872131] Lustre: 147133:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783352885/real 1783352885] req@0000000013ac7225 x1869980982340992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783352901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3052.001019] LustreError: 148917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.012084] LustreError: 148917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3052.864839] LustreError: 148914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.122392] LustreError: 148917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.127654] LustreError: 148917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3058.144109] Lustre: 147133:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783352891/real 1783352891] req@000000004abf96e4 x1869980982342144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783352907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3058.918797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.956695] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3058.962409] Lustre: 148909:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3059.037851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3059.055960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.073953] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3060.498624] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3063.104647] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3064.298207] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3064.305462] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3064.328385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3069.412301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3069.412525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.413340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.416391] LustreError: Skipped 1 previous similar message [ 3069.421112] Lustre: Skipped 4 previous similar messages [ 3071.506802] Lustre: server umount lustre-OST0000 complete [ 3074.530861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.531382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3074.532358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.532363] Lustre: Skipped 1 previous similar message [ 3074.534864] Lustre: Skipped 1 previous similar message [ 3079.649164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.651362] Lustre: Skipped 2 previous similar messages [ 3084.769190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.771942] Lustre: Skipped 1 previous similar message [ 3087.328076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3087.381953] Lustre: server umount lustre-MDT0000 complete [ 3088.597206] LustreError: 149701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783352938 with bad export cookie 800317814926633114 [ 3088.600143] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.601517] LustreError: 149701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3088.713898] Lustre: server umount lustre-MDT0001 complete [ 3092.517292] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 3093.538174] Key type lgssc unregistered [ 3093.674409] LNet: 154520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3093.679118] LNetError: 154520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3093.693327] LNet: Removed LNI 192.168.204.155@tcp [ 3094.008132] Key type .llcrypt unregistered [ 3094.009266] Key type ._llcrypt unregistered [ 3114.555367] Key type ._llcrypt registered [ 3114.557026] Key type .llcrypt registered [ 3114.600432] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3115.092444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3115.113544] alg: No test for adler32 (adler32-zlib) [ 3115.975335] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 3116.078106] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 3117.680212] Key type lgssc registered [ 3118.103917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3121.131337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3121.140115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.306436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.836638] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3127.634991] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3129.606146] LustreError: 156077:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3129.610569] LustreError: 156077:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3129.613188] LustreError: 156077:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3129.720536] Lustre: server umount lustre-MDT0000 complete [ 3134.715950] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3138.007561] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3150.648079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.743758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3150.757807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3150.797947] Lustre: lustre-MDT0000: new disk, initializing [ 3150.824103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.832754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3152.331699] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3156.635937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.678330] Lustre: 159373:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3156.700791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3156.702575] Lustre: Skipped 1 previous similar message [ 3156.735468] Lustre: lustre-MDT0001: new disk, initializing [ 3156.767123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3156.780548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3156.785090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3158.128313] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3161.934985] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3163.825570] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3166.391465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.499273] Lustre: lustre-OST0000: new disk, initializing [ 3166.501513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3166.503807] Lustre: 161342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3166.535155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.529820] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3172.103618] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3173.367545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3173.370195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3173.398555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3178.971035] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3179.568749] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:50:28 (1783353028) [ 3254.240431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3254.242387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.246289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.264617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.264620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.614748] Lustre: server umount lustre-MDT0000 complete [ 3260.384597] LustreError: 159387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.389483] LustreError: 159387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3260.824557] LustreError: 159366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353110 with bad export cookie 15617480966742346919 [ 3260.825351] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.828245] LustreError: 159366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3265.504407] LustreError: 162002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.504468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.518931] Lustre: Skipped 1 previous similar message [ 3265.521569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3265.524979] Lustre: Skipped 2 previous similar messages [ 3270.624437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3270.624438] LustreError: 162760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.232131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3275.314742] Lustre: server umount lustre-MDT0001 complete [ 3282.762125] Lustre: server umount lustre-OST0000 complete [ 3285.667323] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3288.728377] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3300.719538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.800827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3300.812874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3300.850853] Lustre: lustre-MDT0000: new disk, initializing [ 3300.872825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.878748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3302.178488] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3306.131506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.166970] Lustre: 165974:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3306.187307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3306.189519] Lustre: Skipped 1 previous similar message [ 3306.221246] Lustre: lustre-MDT0001: new disk, initializing [ 3306.259306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3306.262412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3307.549558] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3311.282529] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3313.193365] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3315.390652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.487814] Lustre: lustre-OST0000: new disk, initializing [ 3315.490031] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3315.493411] Lustre: 167938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3315.520130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.521908] Lustre: Skipped 1 previous similar message [ 3317.324980] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3319.800315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3319.804601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3319.840398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3320.927315] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3322.243921] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3322.309043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3323.641917] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3323.714788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3324.899378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.899661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.900450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.903025] Lustre: Skipped 1 previous similar message [ 3324.905359] LustreError: Skipped 1 previous similar message [ 3326.416126] Lustre: server umount lustre-OST0000 complete [ 3330.017937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.022475] Lustre: Skipped 1 previous similar message [ 3333.791994] Lustre: server umount lustre-MDT0000 complete [ 3335.138183] LustreError: 165985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.145459] LustreError: 165985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3335.161767] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353185 with bad export cookie 15617480966742505735 [ 3335.162264] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3335.166054] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3335.299518] Lustre: server umount lustre-MDT0001 complete [ 3338.379838] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:53:07 (1783353187) [ 3343.459075] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3346.664432] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3358.387971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.466586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3358.477881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3358.510799] Lustre: lustre-MDT0000: new disk, initializing [ 3358.530705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.536654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.849785] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3363.869499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.911213] Lustre: 172689:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.948810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3363.951342] Lustre: Skipped 3 previous similar messages [ 3364.014248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3365.269415] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3368.838645] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3370.642094] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3372.850257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.957841] Lustre: 174664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3374.949194] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3377.657344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3377.694726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3378.644460] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3382.756084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.756409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3382.757622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3382.761677] Lustre: Skipped 1 previous similar message [ 3382.768507] LustreError: Skipped 1 previous similar message [ 3386.639739] Lustre: server umount lustre-OST0000 complete [ 3389.920821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3402.208120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3402.268751] Lustre: server umount lustre-MDT0000 complete [ 3403.499190] LustreError: 172682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353253 with bad export cookie 15617480966742506603 [ 3403.502292] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.502577] LustreError: 172682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3403.611981] Lustre: server umount lustre-MDT0001 complete [ 3406.445884] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:54:15 (1783353255) [ 3411.832522] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3414.845219] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3426.376321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.452222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3426.461694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3426.463317] Lustre: Skipped 3 previous similar messages [ 3426.496531] Lustre: lustre-MDT0000: new disk, initializing [ 3426.497625] Lustre: Skipped 2 previous similar messages [ 3426.515947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.519147] Lustre: Skipped 2 previous similar messages [ 3426.524621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3426.528593] Lustre: Skipped 2 previous similar messages [ 3427.818291] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3431.849866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.893352] Lustre: 178977:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3432.000102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3433.254705] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3436.754897] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.520332] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3440.882982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.998746] Lustre: 180952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3441.561877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3443.093562] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3446.918107] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3451.875424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3451.875820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.876738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3451.876743] Lustre: Skipped 8 previous similar messages [ 3451.888374] Lustre: Skipped 3 previous similar messages [ 3455.059063] Lustre: server umount lustre-OST0000 complete [ 3462.498810] Lustre: server umount lustre-MDT0000 complete [ 3463.769349] LustreError: 178969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353313 with bad export cookie 15617480966742507688 [ 3463.772814] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.774439] LustreError: 178969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3463.890065] Lustre: server umount lustre-MDT0001 complete [ 3467.277168] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 3468.358851] Key type lgssc unregistered [ 3468.493459] LNet: 182804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3468.495738] LNetError: 182804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3468.505277] LNet: Removed LNI 192.168.204.155@tcp [ 3468.852192] Key type .llcrypt unregistered [ 3468.853973] Key type ._llcrypt unregistered [ 3477.441837] Key type ._llcrypt registered [ 3477.442930] Key type .llcrypt registered [ 3477.487194] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3477.834861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3477.889804] alg: No test for adler32 (adler32-zlib) [ 3478.754542] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 3478.854611] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 3480.440162] Key type lgssc registered [ 3480.848661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3483.532715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3483.540044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.692812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.097094] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3488.885483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3489.001182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3490.399415] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3492.709729] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3494.429436] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.574688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.687718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.579869] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3500.804208] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3511.019759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3524.576957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3524.580251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.585512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.627200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.627795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.632054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.633481] Lustre: Skipped 1 previous similar message [ 3528.912776] Lustre: server umount lustre-OST0000 complete [ 3530.208841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3530.211045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.215455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.217332] Lustre: server umount lustre-MDT0000 complete [ 3536.865096] LustreError: 184195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.870648] LustreError: 184195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3537.445111] LustreError: 184176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353387 with bad export cookie 15098975340872833219 [ 3537.449039] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.449885] LustreError: 184176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3537.561662] Lustre: server umount lustre-MDT0001 complete [ 3541.295018] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 3542.242498] Key type lgssc unregistered [ 3542.362360] LNet: 187744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3542.364718] LNetError: 187744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3542.376322] LNet: Removed LNI 192.168.204.155@tcp [ 3542.683069] Key type .llcrypt unregistered [ 3542.683997] Key type ._llcrypt unregistered [ 3551.636647] Key type ._llcrypt registered [ 3551.638661] Key type .llcrypt registered [ 3551.678309] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3552.090098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3552.132287] alg: No test for adler32 (adler32-zlib) [ 3553.001293] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 3553.099584] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 3554.688141] Key type lgssc registered [ 3555.066455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.725810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3557.731435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.871602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.320570] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3563.316640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.460757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.756635] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3566.994973] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3568.594269] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3570.802030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.919251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.775204] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3575.187113] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.210444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3588.579163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.579960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.580287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.584250] Lustre: Skipped 1 previous similar message [ 3590.984288] Lustre: lustre-OST0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 3590.986425] Lustre: Skipped 1 previous similar message [ 3593.698115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.700353] Lustre: Skipped 1 previous similar message [ 3596.097194] Lustre: lustre-OST0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 3601.217538] Lustre: lustre-OST0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 3601.221576] Lustre: Skipped 2 previous similar messages [ 3602.400131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3602.457539] Lustre: server umount lustre-OST0000 complete [ 3604.888941] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d05b08b7800.ost_server_uuid 50 [ 3605.415556] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d05b08b7800.ost_server_uuid in DISCONN state after 0 sec [ 3605.985823] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3615.901843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.902323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3615.905674] Lustre: Skipped 1 previous similar message [ 3622.035069] Lustre: server umount lustre-MDT0000 complete [ 3623.261379] LustreError: 190976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353473 with bad export cookie 1742802871783182965 [ 3623.264059] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.264386] LustreError: 190976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.374580] Lustre: server umount lustre-MDT0001 complete [ 3626.080383] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:57:54 (1783353474) [ 3628.110530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.242448] LustreError: 193095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3628.246404] LustreError: 193095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3628.268726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.499598] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3632.211464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.318989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.539344] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3635.714244] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3637.412739] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3639.472686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.418604] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3643.826273] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3646.070035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.156080] Lustre: lustre-OST0001: new disk, initializing [ 3646.157688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3646.160064] Lustre: 195931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3646.190985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3646.194236] Lustre: Skipped 1 previous similar message [ 3648.063258] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3649.219525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3649.222494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3649.253951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3651.769696] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3653.089777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3653.091852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.095320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3653.096965] Lustre: Skipped 3 previous similar messages [ 3658.748506] Lustre: server umount lustre-OST0000 complete [ 3659.236033] LustreError: 194914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.243131] LustreError: 194914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3660.803335] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3660.880985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3662.146468] LustreError: 196395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.258013] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3662.336623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3664.354206] LustreError: 194914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.361966] LustreError: 194914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3669.473565] LustreError: 196395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.480072] LustreError: 196395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3674.595554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.419489] Lustre: server umount lustre-OST0001 complete [ 3679.713449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.717034] Lustre: Skipped 2 previous similar messages [ 3685.854587] Lustre: server umount lustre-MDT0000 complete [ 3687.097449] LustreError: 193075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353536 with bad export cookie 1742802871783184029 [ 3687.100545] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3687.101434] LustreError: 193075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3687.210594] Lustre: server umount lustre-MDT0001 complete [ 3695.332803] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:59:04 (1783353544) [ 3697.620885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.705584] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3697.715278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3697.774874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.019306] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3701.760778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.791468] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3701.806395] Lustre: 199342:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3703.155258] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3705.536945] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3707.267751] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3709.407207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.477360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3711.477875] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3713.850198] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3715.595524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3716.070015] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.096105] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3718.084738] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3720.456988] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3722.736798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.739982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3722.744419] Lustre: Skipped 1 previous similar message [ 3722.746273] Lustre: Skipped 9 previous similar messages [ 3722.767978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3727.419811] Lustre: server umount lustre-OST0000 complete [ 3727.845226] LustreError: 202127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.850478] LustreError: 202127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3743.202439] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.202511] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.205832] LustreError: Skipped 1 previous similar message [ 3744.212651] Lustre: server umount lustre-OST0001 complete [ 3748.322810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3751.647639] Lustre: server umount lustre-MDT0000 complete [ 3752.920328] LustreError: 199384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353602 with bad export cookie 1742802871783185191 [ 3752.922742] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.927446] LustreError: 199384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3753.046799] Lustre: server umount lustre-MDT0001 complete [ 3761.097290] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:00:09 (1783353609) [ 3766.437723] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3769.670235] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3782.034298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.138629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3782.158177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3782.224373] Lustre: lustre-MDT0000: new disk, initializing [ 3782.249632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.251833] Lustre: Skipped 3 previous similar messages [ 3782.257753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3783.680148] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3787.967561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.011597] Lustre: 207402:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3788.035833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3788.037661] Lustre: Skipped 1 previous similar message [ 3788.073046] Lustre: lustre-MDT0001: new disk, initializing [ 3788.118113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3788.121877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3789.489393] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3793.343798] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3795.079747] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3797.233235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3797.328487] Lustre: lustre-OST0000: new disk, initializing [ 3797.329985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3797.331932] Lustre: 209368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3798.132882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.136172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3798.170324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3799.151280] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3802.636725] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3803.935279] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3803.994584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3805.269475] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3805.335070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3808.082126] Lustre: server umount lustre-OST0000 complete [ 3813.857523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3813.857747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.859051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.859056] Lustre: Skipped 6 previous similar messages [ 3813.867276] Lustre: Skipped 4 previous similar messages [ 3820.027430] Lustre: server umount lustre-MDT0000 complete [ 3821.255142] LustreError: 208196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353671 with bad export cookie 1742802871783186927 [ 3821.258576] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.258901] LustreError: 208196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3831.826792] Lustre: server umount lustre-MDT0001 complete [ 3834.666943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.837160] LustreError: 211594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.845173] LustreError: 211594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3836.157365] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3838.974172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.492364] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3842.965446] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.811755] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3847.067319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.184166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.186185] Lustre: Skipped 4 previous similar messages [ 3849.191058] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3851.775890] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.310944] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3860.529282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3861.878556] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3861.955236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3863.520841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3868.750923] Lustre: server umount lustre-OST0000 complete [ 3870.793142] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3872.132380] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3895.794202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.904172] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3900.510846] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3901.131939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3904.264735] Lustre: server umount lustre-OST0000 complete [ 3906.016872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3906.021254] LustreError: Skipped 1 previous similar message [ 3906.023766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.030105] Lustre: Skipped 2 previous similar messages [ 3907.607276] Lustre: server umount lustre-MDT0000 complete [ 3908.906726] LustreError: 211575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353758 with bad export cookie 1742802871783187795 [ 3908.910724] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3908.912055] LustreError: 211575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3909.035352] Lustre: server umount lustre-MDT0001 complete [ 3912.030259] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:02:40 (1783353760) [ 3914.326640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.495877] LustreError: 217601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.945882] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3918.833712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.310148] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3922.709867] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3924.625760] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3927.033396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3929.223240] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3931.706491] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3933.272208] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3941.509660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3942.938142] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3943.027864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3945.359632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.435056] Lustre: lustre-OST0001: new disk, initializing [ 3945.437512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3945.441141] Lustre: 221027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3947.479099] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3951.321990] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3952.789210] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3954.176862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3954.180208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3954.211042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3954.905562] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3956.229597] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3956.301901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3959.268319] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.271416] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3959.273314] Lustre: Skipped 9 previous similar messages [ 3963.072825] Lustre: server umount lustre-OST0001 complete [ 3965.028469] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3965.111270] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3966.400751] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3966.471350] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3979.748892] LustreError: 219424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.753827] LustreError: 219424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3990.200349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3990.273717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.275937] Lustre: Skipped 5 previous similar messages [ 3992.305349] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3994.883511] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3995.498113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3995.624249] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3995.628290] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3995.628370] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3995.635793] Lustre: Skipped 1 previous similar message [ 4002.235907] Lustre: server umount lustre-OST0001 complete [ 4020.303535] Lustre: server umount lustre-OST0000 complete [ 4026.337840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4026.340824] LustreError: Skipped 4 previous similar messages [ 4027.664600] Lustre: server umount lustre-MDT0000 complete [ 4028.784355] LustreError: 217582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353878 with bad export cookie 1742802871783188726 [ 4028.787515] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.788682] LustreError: 217582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.232106] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4043.319713] Lustre: server umount lustre-MDT0001 complete [ 4051.468877] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:05:00 (1783353900) [ 4053.822979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.926816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4053.940861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4055.487674] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4058.412575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.450482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4058.460168] Lustre: 226276:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4060.025696] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4062.516054] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4064.414821] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4066.721064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4066.781458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4068.686615] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4070.975242] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4074.618338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.529787] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4078.893329] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4080.435687] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4081.781425] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4081.849703] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4083.112040] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4083.182286] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4084.931873] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d0587f35800.ost_server_uuid 50 [ 4085.398934] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d0587f35800.ost_server_uuid in FULL state after 0 sec [ 4085.735330] Lustre: Permanently deactivating lustre-OST0001 [ 4085.740657] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4086.556570] Lustre: Permanently reactivating lustre-OST0001 [ 4091.877643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.882905] Lustre: Skipped 9 previous similar messages [ 4093.563781] Lustre: server umount lustre-OST0001 complete [ 4109.280129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.334941] Lustre: server umount lustre-OST0000 complete [ 4116.830465] Lustre: server umount lustre-MDT0000 complete [ 4117.473382] LustreError: 227023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.479588] LustreError: 227023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 4118.144288] LustreError: 226266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783353967 with bad export cookie 1742802871783190000 [ 4118.147422] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4118.149275] LustreError: 226266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4118.281623] Lustre: server umount lustre-MDT0001 complete [ 4126.416749] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:06:15 (1783353975) [ 4129.117861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.211813] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4129.226821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4129.230986] Lustre: Skipped 5 previous similar messages [ 4130.787631] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4133.827368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.863403] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4133.865366] Lustre: Skipped 1 previous similar message [ 4133.875670] Lustre: 232959:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4135.420911] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4137.880818] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4139.713614] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4142.028134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.111240] Lustre: Permanently deactivating lustre-OST0000 [ 4142.117045] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4142.838177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4144.060936] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4146.562785] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4148.936846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.147667] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4153.531738] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4157.977224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4159.217332] Lustre: Permanently reactivating lustre-OST0000 [ 4159.225047] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4159.227911] Lustre: Skipped 2 previous similar messages [ 4163.325733] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4163.328294] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.332074] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4163.336083] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4163.339060] LustreError: Skipped 2 previous similar messages [ 4163.350417] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4163.356992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4169.410690] Lustre: server umount lustre-OST0001 complete [ 4185.056163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4185.117834] Lustre: server umount lustre-OST0000 complete [ 4192.600211] Lustre: server umount lustre-MDT0000 complete [ 4193.735230] LustreError: 233697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354043 with bad export cookie 1742802871783191617 [ 4193.738347] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4193.741266] LustreError: 233697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4193.836303] Lustre: server umount lustre-MDT0001 complete [ 4197.439929] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4198.559826] Key type lgssc unregistered [ 4198.689472] LNet: 237808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.691815] LNetError: 237808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.699386] LNet: Removed LNI 192.168.204.155@tcp [ 4199.020260] Key type .llcrypt unregistered [ 4199.021352] Key type ._llcrypt unregistered [ 4208.100546] Key type ._llcrypt registered [ 4208.101650] Key type .llcrypt registered [ 4208.139682] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4208.607862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4208.631085] alg: No test for adler32 (adler32-zlib) [ 4209.486610] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4209.581296] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4211.168105] Key type lgssc registered [ 4211.524521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4214.927480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4214.933942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.091065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.490926] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4220.419588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.462381] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4220.472421] Lustre: 239242:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4220.477624] Lustre: Permanently deactivating lustre-MDT0001 [ 4220.482780] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4220.596438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4221.550263] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4221.552966] LustreError: 239284:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4221.557506] LustreError: 239284:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4221.562207] LustreError: 239284:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4222.032385] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4224.430213] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4226.286750] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4228.555908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.673758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.705509] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4230.755226] LustreError: 241458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.764321] LustreError: 241458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4233.187607] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4235.425183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4235.501789] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4237.508773] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4240.004965] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4240.618093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4240.618543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4240.618987] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4245.772475] Lustre: Permanently reactivating lustre-MDT0001 [ 4245.778196] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4245.782761] Lustre: Skipped 2 previous similar messages [ 4250.441089] Lustre: lustre-MDT0001: Client 2931e41b-acc3-4673-80a5-e23a501c983f (at 192.168.204.55@tcp) reconnecting [ 4252.753767] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4253.356791] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4255.117424] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4255.192981] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4256.615852] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4256.695885] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4257.548793] Lustre: Permanently deactivating lustre-MDT0001 [ 4257.553968] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4257.556417] Lustre: Skipped 2 previous similar messages [ 4265.984971] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4272.097195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4272.099679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.103525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.681509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.682668] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.687694] Lustre: Skipped 2 previous similar messages [ 4277.598699] Lustre: server umount lustre-MDT0000 complete [ 4278.916443] LustreError: 239991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354128 with bad export cookie 3744310798851125051 [ 4278.919863] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.920979] LustreError: 239991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4279.042482] Lustre: server umount lustre-MDT0001 complete [ 4286.394724] Lustre: server umount lustre-OST0000 complete [ 4290.315514] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:08:59 (1783354139) [ 4293.997686] Lustre: server umount lustre-OST0001 complete [ 4296.125265] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 4299.531816] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4311.615890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.702191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4311.705225] Lustre: Skipped 2 previous similar messages [ 4311.713745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4311.754250] Lustre: lustre-MDT0000: new disk, initializing [ 4311.777791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.786125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4313.227179] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4317.434363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.477965] Lustre: 247546:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4317.505619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4317.508206] Lustre: Skipped 1 previous similar message [ 4317.549981] Lustre: lustre-MDT0001: new disk, initializing [ 4317.600320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4317.604813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4318.923702] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4322.650348] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.479305] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4326.787800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.894274] Lustre: lustre-OST0000: new disk, initializing [ 4326.896463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4326.900143] Lustre: 249514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4326.933807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.936188] Lustre: Skipped 1 previous similar message [ 4329.031891] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4332.833502] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4333.049687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4333.053032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4333.082443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4338.495843] LustreError: 250228:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4342.686025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4342.753273] Lustre: lustre-OST0001: new disk, initializing [ 4342.756089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4342.760739] Lustre: 250629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4344.743117] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4348.495739] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4348.584076] LustreError: 250228:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4348.592576] LustreError: 247552:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4349.082745] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4350.465331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4350.469618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4350.500758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4358.688105] LustreError: 247552:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4360.675672] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.679284] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.684929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4360.687865] Lustre: Skipped 4 previous similar messages [ 4365.439511] Lustre: server umount lustre-OST0001 complete [ 4365.797626] LustreError: 249497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.808853] LustreError: 249497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4370.916894] LustreError: 249497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.927966] LustreError: 249497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4376.037374] LustreError: 249496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.042226] LustreError: 249496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4379.105922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.105969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.108467] LustreError: Skipped 2 previous similar messages [ 4379.108903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.112122] Lustre: Skipped 2 previous similar messages [ 4379.116332] Lustre: Skipped 2 previous similar messages [ 4381.156702] LustreError: 249495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.162822] LustreError: 249495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4383.572156] Lustre: server umount lustre-OST0000 complete [ 4386.274061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.274799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.277790] Lustre: Skipped 1 previous similar message [ 4386.279901] Lustre: Skipped 2 previous similar messages [ 4391.072403] Lustre: server umount lustre-MDT0000 complete [ 4391.393039] LustreError: 247556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.397775] LustreError: 247556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4392.340730] LustreError: 249513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354242 with bad export cookie 3744310798851127641 [ 4392.345538] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.345953] LustreError: 249513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4398.515567] Lustre: server umount lustre-MDT0001 complete [ 4402.429489] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4403.514343] Key type lgssc unregistered [ 4403.647505] LNet: 252664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4403.650324] LNetError: 252664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4403.660420] LNet: Removed LNI 192.168.204.155@tcp [ 4403.976452] Key type .llcrypt unregistered [ 4403.978095] Key type ._llcrypt unregistered [ 4418.674155] Key type ._llcrypt registered [ 4418.675355] Key type .llcrypt registered [ 4418.718240] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4419.298540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4419.310293] alg: No test for adler32 (adler32-zlib) [ 4420.186907] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4420.289407] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4421.880143] Key type lgssc registered [ 4422.317781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4425.044524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4425.052548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.142332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4426.149779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4426.213761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.629233] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4430.477405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.510558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4430.522198] Lustre: 254847:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4430.626409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.013530] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4434.674145] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4436.527384] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4438.770948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.843587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4438.911986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.856579] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4443.233244] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4445.710135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4491.749997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.750339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.751661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.752732] LustreError: Skipped 1 previous similar message [ 4491.756197] Lustre: Skipped 1 previous similar message [ 4493.287219] Lustre: server umount lustre-OST0000 complete [ 4494.751662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4498.860171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4499.020155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4500.963229] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4503.397761] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4511.266430] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.270954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.271051] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4511.279906] Lustre: Skipped 1 previous similar message [ 4536.801600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.803522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.805083] LustreError: Skipped 1 previous similar message [ 4536.809485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4536.812800] Lustre: Skipped 2 previous similar messages [ 4541.923555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.928131] Lustre: Skipped 1 previous similar message [ 4542.876223] Lustre: server umount lustre-OST0000 complete [ 4547.042161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.042792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.046564] Lustre: Skipped 2 previous similar messages [ 4547.050731] Lustre: Skipped 1 previous similar message [ 4550.302363] Lustre: server umount lustre-MDT0000 complete [ 4551.539330] LustreError: 254840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354401 with bad export cookie 10297901762389044677 [ 4551.542964] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.543386] LustreError: 254840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4551.656208] Lustre: server umount lustre-MDT0001 complete [ 4555.558373] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4556.618199] Key type lgssc unregistered [ 4556.749430] LNet: 260258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4556.751839] LNetError: 260258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4556.761324] LNet: Removed LNI 192.168.204.155@tcp [ 4557.088150] Key type .llcrypt unregistered [ 4557.089791] Key type ._llcrypt unregistered [ 4566.207989] Key type ._llcrypt registered [ 4566.209021] Key type .llcrypt registered [ 4566.248647] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4566.689160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4566.700661] alg: No test for adler32 (adler32-zlib) [ 4567.554328] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4567.647647] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4569.232131] Key type lgssc registered [ 4569.593469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4572.177240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4572.185043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.338199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4574.708730] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4577.620583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.752193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4579.131264] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4581.527914] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4583.357842] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4585.686591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.806709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.795707] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4590.289101] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4599.144553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4608.481810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4608.484403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.488980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4609.507866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4609.508340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.508387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4613.971886] Lustre: server umount lustre-OST0000 complete [ 4618.720872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4618.723604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.728513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.730741] Lustre: Skipped 1 previous similar message [ 4621.397493] Lustre: server umount lustre-MDT0000 complete [ 4622.636164] LustreError: 261632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354472 with bad export cookie 14968903150223648204 [ 4622.639076] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.639713] LustreError: 261632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4622.757268] Lustre: server umount lustre-MDT0001 complete [ 4626.430280] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4627.461332] Key type lgssc unregistered [ 4627.588395] LNet: 265970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4627.591923] LNetError: 265970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4627.599351] LNet: Removed LNI 192.168.204.155@tcp [ 4627.915696] Key type .llcrypt unregistered [ 4627.917560] Key type ._llcrypt unregistered [ 4635.118941] Key type ._llcrypt registered [ 4635.120610] Key type .llcrypt registered [ 4635.163201] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4635.496066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4635.525986] alg: No test for adler32 (adler32-zlib) [ 4636.382953] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4636.475120] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4638.056196] Key type lgssc registered [ 4638.446369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4641.054225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4641.060733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.221365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.639467] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4646.538273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.664335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.073724] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4650.470652] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4652.277761] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4654.563717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.684350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.642340] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4659.115978] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.004087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4673.506667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4673.507561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.508618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.522742] Lustre: Skipped 1 previous similar message [ 4678.626045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4678.628757] Lustre: Skipped 2 previous similar messages [ 4679.501157] Lustre: server umount lustre-OST0000 complete [ 4682.720835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4682.723371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.728426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4683.746246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.935182] Lustre: server umount lustre-MDT0000 complete [ 4688.192356] LustreError: 267151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354538 with bad export cookie 18050991272501181070 [ 4688.194276] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.199285] LustreError: 267151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4688.316688] Lustre: server umount lustre-MDT0001 complete [ 4691.885512] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4692.935348] Key type lgssc unregistered [ 4693.068494] LNet: 270865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4693.071394] LNetError: 270865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4693.080342] LNet: Removed LNI 192.168.204.155@tcp [ 4693.400574] Key type .llcrypt unregistered [ 4693.401968] Key type ._llcrypt unregistered [ 4702.358712] Key type ._llcrypt registered [ 4702.359928] Key type .llcrypt registered [ 4702.406319] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4703.009614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4703.017915] alg: No test for adler32 (adler32-zlib) [ 4703.890533] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4703.998483] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4705.592196] Key type lgssc registered [ 4706.014743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4708.702096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4708.708158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.846435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4711.177819] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4713.985711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.113038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4715.427837] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4717.763405] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4719.521083] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4721.717633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4721.843092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4723.766024] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4726.154633] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4735.144079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4745.698620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4745.698620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.699782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4745.705893] Lustre: Skipped 1 previous similar message [ 4746.897714] Lustre: server umount lustre-OST0000 complete [ 4750.229857] Lustre: server umount lustre-MDT0000 complete [ 4750.304794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4750.307329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.310724] Lustre: Skipped 1 previous similar message [ 4750.312569] LustreError: 272272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.317263] LustreError: 272272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4750.817857] LustreError: 272268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.482271] LustreError: 272255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354601 with bad export cookie 15513122292854992124 [ 4751.485690] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.487943] LustreError: 272255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4751.598720] Lustre: server umount lustre-MDT0001 complete [ 4755.232709] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4756.340419] Key type lgssc unregistered [ 4756.475461] LNet: 276592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4756.478657] LNetError: 276592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4756.488322] LNet: Removed LNI 192.168.204.155@tcp [ 4756.885712] Key type .llcrypt unregistered [ 4756.887563] Key type ._llcrypt unregistered [ 4764.093008] Key type ._llcrypt registered [ 4764.094047] Key type .llcrypt registered [ 4764.129654] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4764.583309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4764.592702] alg: No test for adler32 (adler32-zlib) [ 4765.459621] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4765.547893] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4767.128141] Key type lgssc registered [ 4767.495247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4770.070217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4770.075655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.239494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.619718] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4775.867015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.991046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4777.379193] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4779.801298] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4781.568440] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4783.780079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4783.909762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.755509] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4788.174331] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.204238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4806.626111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4806.628248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.629275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4806.631467] Lustre: Skipped 1 previous similar message [ 4808.334581] Lustre: server umount lustre-OST0000 complete [ 4811.745861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4811.745929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.746996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.747020] Lustre: Skipped 1 previous similar message [ 4811.749401] LustreError: Skipped 1 previous similar message [ 4811.753543] Lustre: Skipped 1 previous similar message [ 4815.707566] Lustre: server umount lustre-MDT0000 complete [ 4816.865046] LustreError: 277813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.869588] LustreError: 277813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4816.921127] LustreError: 277769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354666 with bad export cookie 5433153302667766012 [ 4816.924360] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.926487] LustreError: 277769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4817.050430] Lustre: server umount lustre-MDT0001 complete [ 4820.793618] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4821.791872] Key type lgssc unregistered [ 4821.915361] LNet: 281519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4821.917632] LNetError: 281519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4821.930241] LNet: Removed LNI 192.168.204.155@tcp [ 4822.220975] Key type .llcrypt unregistered [ 4822.222284] Key type ._llcrypt unregistered [ 4825.663364] Key type ._llcrypt registered [ 4825.664409] Key type .llcrypt registered [ 4825.708848] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4831.224669] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 4837.675636] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4838.221870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4838.230836] alg: No test for adler32 (adler32-zlib) [ 4839.100082] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4839.203731] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4840.792129] Key type lgssc registered [ 4841.202401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4852.434387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4852.440033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.530616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4853.543814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4853.585401] Lustre: lustre-MDT0000: new disk, initializing [ 4853.606247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.613495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4855.029389] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4859.379839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.426963] Lustre: 285041:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4859.456777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4859.459924] Lustre: Skipped 1 previous similar message [ 4859.499644] Lustre: lustre-MDT0001: new disk, initializing [ 4859.536821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.549234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4859.553523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4860.880165] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4864.542443] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4866.330489] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4868.618035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4868.733824] Lustre: lustre-OST0000: new disk, initializing [ 4868.735629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4868.738079] Lustre: 287009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4868.769638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.808269] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4874.448878] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4875.917141] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4877.313940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4877.317478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4877.351831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4878.026861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4879.447256] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4879.528727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4882.403835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.404059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4882.410519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4882.411907] LustreError: Skipped 1 previous similar message [ 4882.417893] Lustre: Skipped 1 previous similar message [ 4886.289531] Lustre: server umount lustre-OST0000 complete [ 4890.593292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4890.596235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.600632] Lustre: Skipped 1 previous similar message [ 4890.602686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.641231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.641800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.728566] Lustre: server umount lustre-MDT0000 complete [ 4894.990387] LustreError: 285034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354744 with bad export cookie 1664593722561352020 [ 4894.993548] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.994577] LustreError: 285034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4895.120865] Lustre: server umount lustre-MDT0001 complete [ 4898.088295] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:19:06 (1783354746) [ 4900.358443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.570650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4901.941851] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4904.729260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.859348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4906.250810] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4908.662324] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4910.415521] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.683396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4914.783376] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4917.122301] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4931.557255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.558432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4931.559084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4931.559091] Lustre: Skipped 1 previous similar message [ 4931.560667] Lustre: Skipped 1 previous similar message [ 4936.674955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.676606] Lustre: Skipped 2 previous similar messages [ 4936.784418] Lustre: server umount lustre-OST0000 complete [ 4940.768804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.772147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.772244] LustreError: Skipped 1 previous similar message [ 4944.277212] Lustre: server umount lustre-MDT0000 complete [ 4945.494263] LustreError: 289411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354795 with bad export cookie 1664593722561352888 [ 4945.496788] LustreError: 290167:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4945.497243] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.499707] LustreError: 289411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4945.617879] Lustre: server umount lustre-MDT0001 complete [ 4949.420737] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4950.422874] Key type lgssc unregistered [ 4950.545563] LNet: 292982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4950.550444] LNetError: 292982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4950.560436] LNet: Removed LNI 192.168.204.155@tcp [ 4950.914248] Key type .llcrypt unregistered [ 4950.915331] Key type ._llcrypt unregistered [ 4959.779662] Key type ._llcrypt registered [ 4959.780977] Key type .llcrypt registered [ 4959.825287] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 4966.021306] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4966.435837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4966.523718] alg: No test for adler32 (adler32-zlib) [ 4967.377738] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 4967.468910] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4969.048147] Key type lgssc registered [ 4969.467903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4984.136544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4984.142620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.229554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4985.241818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4985.290823] Lustre: lustre-MDT0000: new disk, initializing [ 4985.333313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4985.341778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4986.733815] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4990.936787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.983081] Lustre: 296778:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4991.010497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4991.012286] Lustre: Skipped 1 previous similar message [ 4991.049381] Lustre: lustre-MDT0001: new disk, initializing [ 4991.087916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4991.103525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4991.106948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4992.524296] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4996.314091] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4998.129283] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5000.441446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5000.558082] Lustre: lustre-OST2710: new disk, initializing [ 5000.560878] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5000.565702] Lustre: 298748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5000.596148] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5002.579326] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5006.307646] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5006.840121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5006.844760] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5006.878265] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5008.637835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.706069] Lustre: lustre-OST03e8: new disk, initializing [ 5008.707698] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5008.710093] Lustre: 299808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5008.740266] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5010.735851] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5011.982846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5011.988165] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5012.016577] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5013.194948] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5015.892596] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5015.964204] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5017.311546] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5017.382403] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5018.784813] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5018.864995] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5020.190433] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5020.258667] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5022.181446] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.182237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.187536] Lustre: Skipped 3 previous similar messages [ 5022.189462] Lustre: Skipped 2 previous similar messages [ 5022.193838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5027.299253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5027.300996] Lustre: Skipped 1 previous similar message [ 5027.373389] Lustre: server umount lustre-MDT0000 complete [ 5028.625724] LustreError: 296770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354878 with bad export cookie 3058386417135652782 [ 5028.631637] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.767277] Lustre: server umount lustre-MDT0001 complete [ 5039.622479] Lustre: server umount lustre-OST2710 complete [ 5051.035487] Lustre: server umount lustre-OST03e8 complete [ 5057.004175] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 5060.210325] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5071.749596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.834664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5071.844775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5071.885378] Lustre: lustre-MDT0000: new disk, initializing [ 5071.909105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.914728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5073.197741] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5077.286826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.327320] Lustre: 305049:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5077.438280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5078.780434] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5082.356590] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5084.056058] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5086.271851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.387076] Lustre: lustre-OST0000: new disk, initializing [ 5086.388633] Lustre: Skipped 1 previous similar message [ 5086.390399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5086.392494] Lustre: Skipped 2 previous similar messages [ 5086.394603] Lustre: 307017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5086.427864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.429933] Lustre: Skipped 1 previous similar message [ 5087.482184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5087.485854] Lustre: Skipped 1 previous similar message [ 5087.488100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5087.520218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5088.374796] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5092.067436] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.478568] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5093.552625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5094.878187] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5094.966066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5097.955140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.956180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.958824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5097.961636] Lustre: Skipped 1 previous similar message [ 5097.964886] Lustre: Skipped 4 previous similar messages [ 5101.719210] Lustre: server umount lustre-OST0000 complete [ 5103.073810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.073958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.074712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.076628] LustreError: Skipped 1 previous similar message [ 5103.080235] Lustre: Skipped 1 previous similar message [ 5108.193326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.197027] Lustre: Skipped 2 previous similar messages [ 5109.088675] Lustre: server umount lustre-MDT0000 complete [ 5110.352306] LustreError: 305042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783354960 with bad export cookie 3058386417135654448 [ 5110.356283] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5110.357294] LustreError: 305042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5110.488858] Lustre: server umount lustre-MDT0001 complete [ 5113.193485] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:22:42 (1783354962) [ 5113.689560] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5114.244004] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:22:43 (1783354963) [ 5115.313863] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5124.562991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.665205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5124.674318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5124.733075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.128292] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5129.039978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.076574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5129.089611] Lustre: 310870:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.653471] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5146.530867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5146.603742] Lustre: Denying initial registration attempt from nid 192.168.204.155@tcp, specified as failover [ 5146.606670] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5146.609834] LustreError: 312323:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5146.611932] LustreError: 312323:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5146.613808] LustreError: 312323:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5146.635750] Lustre: server umount lustre-OST0000 complete [ 5146.637075] LustreError: 312323:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5149.664968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5149.667158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.670783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5153.762083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.840055] Lustre: server umount lustre-MDT0000 complete [ 5156.119111] LustreError: 311610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355005 with bad export cookie 3058386417135655316 [ 5156.122028] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.127649] LustreError: 311610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5156.247577] Lustre: server umount lustre-MDT0001 complete [ 5159.923231] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5160.963751] Key type lgssc unregistered [ 5161.098446] LNet: 313442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.102365] LNetError: 313442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.112314] LNet: Removed LNI 192.168.204.155@tcp [ 5161.438853] Key type .llcrypt unregistered [ 5161.439984] Key type ._llcrypt unregistered [ 5165.112175] Key type ._llcrypt registered [ 5165.113255] Key type .llcrypt registered [ 5165.151293] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5165.514990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5165.572737] alg: No test for adler32 (adler32-zlib) [ 5166.431303] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5166.529988] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5168.120165] Key type lgssc registered [ 5168.477074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.339042] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5195.508019] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5195.514294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.615052] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5196.623396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.697499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5198.150813] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5200.962053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.000561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5201.013664] Lustre: 316177:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5201.109702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.490933] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5204.871151] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5206.664482] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5221.922858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5221.996973] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5222.068243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.835788] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5226.128419] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5226.875168] Lustre: server umount lustre-OST0000 complete [ 5231.087205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.088105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.092735] Lustre: Skipped 1 previous similar message [ 5236.193017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.194832] Lustre: Skipped 2 previous similar messages [ 5240.289385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5240.291283] Lustre: Skipped 1 previous similar message [ 5242.336099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5242.399148] Lustre: server umount lustre-MDT0000 complete [ 5243.597195] LustreError: 316919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355093 with bad export cookie 6476501470455315396 [ 5243.600132] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5243.601963] LustreError: 316919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5243.724135] Lustre: server umount lustre-MDT0001 complete [ 5247.147445] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5248.088404] Key type lgssc unregistered [ 5248.216272] LNet: 319742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.218588] LNetError: 319742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5248.226243] LNet: Removed LNI 192.168.204.155@tcp [ 5248.514156] Key type .llcrypt unregistered [ 5248.515353] Key type ._llcrypt unregistered [ 5257.083570] Key type ._llcrypt registered [ 5257.084629] Key type .llcrypt registered [ 5257.120423] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5257.644907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5257.666046] alg: No test for adler32 (adler32-zlib) [ 5258.536547] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5258.633033] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5260.216152] Key type lgssc registered [ 5260.611776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5263.147401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5263.152780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.298635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.703423] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5268.599816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.734361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5270.172912] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5272.513097] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5274.200780] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5278.306926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5278.413510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.151795] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5282.376671] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5299.018383] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 5299.171430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.175202] Lustre: Skipped 2 previous similar messages [ 5301.916667] Lustre: server umount lustre-MDT0000 complete [ 5303.096077] LustreError: 321117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355152 with bad export cookie 17848930379458915437 [ 5303.100506] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.101255] LustreError: 321117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5304.129283] LustreError: 321130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.134304] LustreError: 321130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5304.134923] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 5304.138770] Lustre: Skipped 3 previous similar messages [ 5304.289289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.320728] LustreError: 321130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.320959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5308.326403] LustreError: 321130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5308.331232] Lustre: Skipped 1 previous similar message [ 5309.291656] Lustre: server umount lustre-MDT0001 complete [ 5311.317013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5313.992861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.840750] LustreError: 320445:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000550380ee x1869983365750784/t0(0) o250->MGC192.168.204.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5327.970456] LustreError: 322965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.976024] LustreError: 322965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5327.998522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.021469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5329.342294] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5332.225105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.376269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5333.413898] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5333.685608] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5336.043600] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5337.509453] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5339.628795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5341.356428] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5341.882625] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5343.713287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.715845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.720292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5347.810333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5347.811380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5347.814604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.204891] Lustre: server umount lustre-OST0000 complete [ 5352.930088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.930562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.939466] Lustre: Skipped 1 previous similar message [ 5356.633924] Lustre: server umount lustre-MDT0000 complete [ 5357.827160] LustreError: 321161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355207 with bad export cookie 17848930379458922591 [ 5357.829804] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.830490] LustreError: 321161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5357.950157] Lustre: server umount lustre-MDT0001 complete [ 5361.451634] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5362.474939] Key type lgssc unregistered [ 5362.617485] LNet: 327421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5362.619939] LNetError: 327421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5362.627384] LNet: Removed LNI 192.168.204.155@tcp [ 5362.928542] Key type .llcrypt unregistered [ 5362.930218] Key type ._llcrypt unregistered [ 5371.819884] Key type ._llcrypt registered [ 5371.821774] Key type .llcrypt registered [ 5371.865779] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5372.211957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5372.220427] alg: No test for adler32 (adler32-zlib) [ 5373.076294] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5373.173671] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5374.752192] Key type lgssc registered [ 5375.195610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5377.936024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5377.944311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.117676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.509212] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5383.355209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.482770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5384.703123] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5387.068418] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5388.820745] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5393.890158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5393.890417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.896775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.898497] Lustre: Skipped 1 previous similar message [ 5399.010938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.015443] Lustre: Skipped 1 previous similar message [ 5404.128178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5404.129885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.136450] Lustre: Skipped 1 previous similar message [ 5404.204253] Lustre: server umount lustre-MDT0000 complete [ 5405.646293] LustreError: 330426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355255 with bad export cookie 12452577191262406812 [ 5405.650521] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5405.652341] LustreError: 330426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5405.778167] Lustre: server umount lustre-MDT0001 complete [ 5410.737960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.830749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5410.841224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5410.915799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.218656] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5415.093831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.133649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5415.149206] Lustre: 331398:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.268698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.856481] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5419.347801] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5421.236498] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5425.892526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5425.962727] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5425.966416] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5425.969784] LustreError: 333278:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5425.972350] LustreError: 333278:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5425.974380] LustreError: 333278:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5425.998615] Lustre: server umount lustre-OST0000 complete [ 5425.999941] LustreError: 333278:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5430.819160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5430.901624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5430.973967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5432.906979] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5435.235585] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5437.871385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.907099] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5437.935277] Lustre: lustre-OST0001: new disk, initializing [ 5437.937316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5437.939918] Lustre: 334808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5439.897955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5439.903522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5439.932962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5439.970432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5440.041516] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5442.372736] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5445.092153] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.093495] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.095140] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5445.098353] Lustre: Skipped 2 previous similar messages [ 5449.410284] Lustre: server umount lustre-OST0001 complete [ 5450.212622] LustreError: 333746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.217468] LustreError: 333746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5455.331125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5455.331164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.333179] LustreError: 333745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.333536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.333543] Lustre: Skipped 1 previous similar message [ 5455.334127] LustreError: Skipped 2 previous similar messages [ 5455.348620] Lustre: Skipped 1 previous similar message [ 5456.785882] Lustre: server umount lustre-OST0000 complete [ 5460.449696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.167088] Lustre: server umount lustre-MDT0000 complete [ 5465.420179] LustreError: 331390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355315 with bad export cookie 12452577191262407351 [ 5465.422658] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5465.423990] LustreError: 331390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5465.541385] Lustre: server umount lustre-MDT0001 complete [ 5469.259259] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5470.245300] Key type lgssc unregistered [ 5470.369404] LNet: 336764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.372364] LNetError: 336764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.380290] LNet: Removed LNI 192.168.204.155@tcp [ 5470.698405] Key type .llcrypt unregistered [ 5470.699714] Key type ._llcrypt unregistered [ 5502.545747] Key type ._llcrypt registered [ 5502.547040] Key type .llcrypt registered [ 5502.586968] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 5508.667986] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5508.977286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5509.005027] alg: No test for adler32 (adler32-zlib) [ 5509.855019] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5509.941954] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5511.520181] Key type lgssc registered [ 5511.917696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5524.077266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5524.085890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.167793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5525.178103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5525.213456] Lustre: lustre-MDT0000: new disk, initializing [ 5525.235927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.242218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5526.691978] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5530.730093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.770108] Lustre: 341664:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5530.795929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5530.798045] Lustre: Skipped 1 previous similar message [ 5530.833073] Lustre: lustre-MDT0001: new disk, initializing [ 5530.863512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.876754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5530.881566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5532.233535] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5535.975521] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5537.735521] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5539.948431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5540.067398] Lustre: lustre-OST0000: new disk, initializing [ 5540.070213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5540.073345] Lustre: 343633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5540.110168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.446431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5540.449606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5540.494656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5542.009329] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5545.697444] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5547.096193] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5547.168936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5548.671679] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5548.749049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5551.078173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5551.078871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.082878] LustreError: Skipped 1 previous similar message [ 5551.087742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5551.092625] Lustre: Skipped 1 previous similar message [ 5551.569720] Lustre: server umount lustre-OST0000 complete [ 5556.194413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.195152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5556.199510] Lustre: Skipped 2 previous similar messages [ 5559.033634] Lustre: server umount lustre-MDT0000 complete [ 5560.573072] LustreError: 341659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355410 with bad export cookie 3919840853948010049 [ 5560.578614] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5560.581053] LustreError: 341659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5560.727249] Lustre: server umount lustre-MDT0001 complete [ 5563.496611] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:30:12 (1783355412) [ 5566.098065] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:30:14 (1783355414) [ 5569.099275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.291368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.511326] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5574.005598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.167268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5575.572630] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5577.997681] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5580.058224] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.282345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.388439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.227085] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5586.711429] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5600.524141] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5601.060653] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5601.638709] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5602.297303] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5603.024180] Lustre: Failing over lustre-MDT0000 [ 5603.216441] Lustre: server umount lustre-MDT0000 complete [ 5604.833668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.840125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.851878] LustreError: 346385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.862358] LustreError: 346385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5605.857514] LustreError: 346384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.862911] LustreError: 346384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5607.247753] LustreError: 346379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.978128] LustreError: 346403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.989960] LustreError: 346403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5616.097385] LustreError: 346385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.103478] LustreError: 346385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5616.180167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.216801] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.299453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.327069] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5617.472820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5617.635565] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5621.735198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5621.745431] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5621.761848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5623.581138] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5624.178523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5626.026359] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5626.694902] Lustre: Failing over lustre-MDT0000 [ 5626.850960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5626.851088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.855792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.860432] Lustre: Skipped 4 previous similar messages [ 5626.863210] Lustre: Skipped 3 previous similar messages [ 5626.934195] Lustre: server umount lustre-MDT0000 complete [ 5627.722712] LustreError: 346380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.221689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.290428] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.432657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5642.145320] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5643.075285] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5645.808069] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5645.813818] Lustre: Skipped 2 previous similar messages [ 5645.818812] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5645.851882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5647.770928] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5648.281295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5650.917419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.918939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5650.920163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5650.922688] Lustre: Skipped 1 previous similar message [ 5656.034447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.040611] Lustre: Skipped 4 previous similar messages [ 5656.887355] Lustre: server umount lustre-MDT0000 complete [ 5658.405838] LustreError: 346365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355508 with bad export cookie 3919840853948012730 [ 5658.411233] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.414157] LustreError: 346365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5658.551207] Lustre: server umount lustre-MDT0001 complete [ 5669.276102] Lustre: server umount lustre-OST0000 complete [ 5675.034603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.192338] LustreError: 352733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.198740] LustreError: 352733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5675.213384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.215423] Lustre: Skipped 1 previous similar message [ 5676.520332] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5679.628742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.037391] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5683.391148] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5685.434220] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5687.917877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.468782] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5693.392307] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5701.416974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5705.697491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.701462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.708979] Lustre: Skipped 1 previous similar message [ 5705.712213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.472462] Lustre: server umount lustre-OST0000 complete [ 5710.870542] Lustre: server umount lustre-MDT0000 complete [ 5711.842110] LustreError: 352733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.854038] LustreError: 352733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5712.198401] LustreError: 352715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355562 with bad export cookie 3919840853948013451 [ 5712.202348] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.205713] LustreError: 352715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5712.323641] Lustre: server umount lustre-MDT0001 complete [ 5716.038719] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5717.370279] Key type lgssc unregistered [ 5717.509370] LNet: 356282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5717.512275] LNetError: 356282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5717.522244] LNet: Removed LNI 192.168.204.155@tcp [ 5717.877757] Key type .llcrypt unregistered [ 5717.880057] Key type ._llcrypt unregistered [ 5728.585966] Key type ._llcrypt registered [ 5728.587063] Key type .llcrypt registered [ 5728.622432] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5729.070525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5729.096047] alg: No test for adler32 (adler32-zlib) [ 5729.943683] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5730.029768] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5731.608132] Key type lgssc registered [ 5732.189763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.662196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5735.672310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.831901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.237612] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5741.778111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.939165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5743.598904] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5746.584681] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5748.616590] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.111366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.264780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.480602] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5756.148892] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5761.519389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5771.746997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.750485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.754741] Lustre: Skipped 2 previous similar messages [ 5771.759834] Lustre: Skipped 2 previous similar messages [ 5776.799497] Lustre: server umount lustre-MDT0000 complete [ 5776.865895] LustreError: 357775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.870982] LustreError: 357775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5778.091181] LustreError: 357757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355627 with bad export cookie 14233978666762162230 [ 5778.094092] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.094842] LustreError: 357757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5778.195939] Lustre: server umount lustre-MDT0001 complete [ 5789.235748] Lustre: server umount lustre-OST0000 complete [ 5794.457418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5794.608578] LustreError: 361505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.630499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.080863] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5799.455169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.600440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5801.172984] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5803.817996] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5805.726535] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5808.604174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.747273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.176715] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5814.538234] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.010311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5829.090001] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.091242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.093430] Lustre: Skipped 2 previous similar messages [ 5829.096649] Lustre: Skipped 2 previous similar messages [ 5833.383984] Lustre: server umount lustre-MDT0000 complete [ 5834.209869] LustreError: 361505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.216096] LustreError: 361505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5834.840367] LustreError: 361486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355684 with bad export cookie 14233978666762163329 [ 5834.845726] LustreError: 361486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5834.847040] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.984523] Lustre: server umount lustre-MDT0001 complete [ 5846.550838] Lustre: server umount lustre-OST0000 complete [ 5853.161489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.321661] LustreError: 365411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.343185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5854.588125] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5857.321307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.710953] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5861.265219] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5863.286796] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5865.676331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.063024] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5871.000676] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5879.152402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5883.362581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.367314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.375590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5884.389362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5885.102668] Lustre: server umount lustre-OST0000 complete [ 5888.480991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5888.483519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.488201] Lustre: Skipped 1 previous similar message [ 5888.489964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5888.491872] Lustre: Skipped 2 previous similar messages [ 5888.920666] Lustre: server umount lustre-MDT0000 complete [ 5889.505496] LustreError: 365430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.511557] LustreError: 365430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5890.198204] LustreError: 365392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355740 with bad export cookie 14233978666762164337 [ 5890.201071] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.201667] LustreError: 365392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5890.312368] Lustre: server umount lustre-MDT0001 complete [ 5894.685518] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5896.043147] Key type lgssc unregistered [ 5896.176459] LNet: 368961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5896.179718] LNetError: 368961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5896.189608] LNet: Removed LNI 192.168.204.155@tcp [ 5896.546660] Key type .llcrypt unregistered [ 5896.547961] Key type ._llcrypt unregistered [ 5905.721805] Key type ._llcrypt registered [ 5905.723883] Key type .llcrypt registered [ 5905.769245] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5906.079423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5906.085269] alg: No test for adler32 (adler32-zlib) [ 5906.929989] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5907.011543] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5908.592192] Key type lgssc registered [ 5908.919306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5911.283164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5911.288543] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.291230] LustreError: 370377:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5911.299118] LustreError: 370377:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5911.302842] LustreError: 370377:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5911.305784] LustreError: 370377:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5911.309968] LustreError: 370377:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5913.821617] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.824169] LustreError: 370775:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5913.832882] LustreError: 370775:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5913.835171] LustreError: 370775:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5913.838351] LustreError: 370775:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5913.841864] LustreError: 370775:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5920.400747] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5921.349922] Key type lgssc unregistered [ 5921.468431] LNet: 371775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5921.471212] LNetError: 371775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5921.480391] LNet: Removed LNI 192.168.204.155@tcp [ 5921.769411] Key type .llcrypt unregistered [ 5921.771029] Key type ._llcrypt unregistered [ 5929.063370] Key type ._llcrypt registered [ 5929.064896] Key type .llcrypt registered [ 5929.107853] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 5935.056521] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5935.375926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5935.440460] alg: No test for adler32 (adler32-zlib) [ 5936.292267] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 5936.382888] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5937.968231] Key type lgssc registered [ 5938.448447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5951.445251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5951.450353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.545770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5952.557885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5952.598696] Lustre: lustre-MDT0000: new disk, initializing [ 5952.626063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.634205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5953.929718] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5959.058644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.126964] Lustre: 374902:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5959.157879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5959.161165] Lustre: Skipped 1 previous similar message [ 5959.199547] Lustre: lustre-MDT0001: new disk, initializing [ 5959.238349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5959.252166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5959.256806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5960.986757] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5965.033476] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5966.773972] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5969.389715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5969.541161] Lustre: lustre-OST0000: new disk, initializing [ 5969.545391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5969.551254] Lustre: 376871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5969.605091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.194402] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5976.592760] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5977.099337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5977.108337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5977.160722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5978.307663] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5978.384946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5979.914473] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5979.984356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5982.182941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.183515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5982.193313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5982.197309] LustreError: Skipped 1 previous similar message [ 5986.796748] Lustre: server umount lustre-OST0000 complete [ 5989.857238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.860921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.864774] Lustre: Skipped 1 previous similar message [ 5989.866525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.868299] Lustre: Skipped 1 previous similar message [ 5992.418137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.419182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.716318] Lustre: server umount lustre-MDT0000 complete [ 5996.060658] LustreError: 375698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355845 with bad export cookie 10834908520733196347 [ 5996.064863] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.065017] LustreError: 375698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5996.193382] Lustre: server umount lustre-MDT0001 complete [ 5999.129487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5999.837739] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:37:28 (1783355848) [ 6002.402320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.579964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.833651] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6007.028847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.148684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6008.549393] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6011.247980] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6012.932151] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6015.112308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.217220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.555427] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6020.296829] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6023.022961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.082332] Lustre: lustre-OST0001: new disk, initializing [ 6023.084571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6023.087393] Lustre: 382166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6025.481933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6025.485360] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6025.485785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6025.523928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6029.636684] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6032.875467] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.878338] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6032.884626] Lustre: Skipped 1 previous similar message [ 6032.888955] Lustre: Skipped 1 previous similar message [ 6037.629080] Lustre: server umount lustre-OST0001 complete [ 6037.989882] LustreError: 381152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.000170] LustreError: 381152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6043.109326] LustreError: 382529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.114072] LustreError: 382529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6048.227848] LustreError: 381151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.232829] LustreError: 381151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6052.321361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6052.325419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.330647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6052.333432] Lustre: Skipped 1 previous similar message [ 6053.347432] LustreError: 381151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.351979] LustreError: 381151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6058.473259] LustreError: 381153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.482323] LustreError: 381153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6063.590675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.592349] Lustre: Skipped 5 previous similar messages [ 6064.096111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6064.172827] Lustre: server umount lustre-OST0000 complete [ 6068.709299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.716960] Lustre: Skipped 2 previous similar messages [ 6068.720313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6068.725251] LustreError: Skipped 1 previous similar message [ 6071.784508] Lustre: server umount lustre-MDT0000 complete [ 6073.064731] LustreError: 382165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355922 with bad export cookie 10834908520733197215 [ 6073.070492] LustreError: 382165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6073.070739] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.215912] Lustre: server umount lustre-MDT0001 complete [ 6083.182145] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6084.481305] Key type lgssc unregistered [ 6084.605318] LNet: 384150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6084.607618] LNetError: 384150:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6084.615282] LNet: Removed LNI 192.168.204.155@tcp [ 6084.926384] Key type .llcrypt unregistered [ 6084.927912] Key type ._llcrypt unregistered [ 6106.208285] Key type ._llcrypt registered [ 6106.209587] Key type .llcrypt registered [ 6106.248264] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6106.635031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6106.652188] alg: No test for adler32 (adler32-zlib) [ 6107.499315] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6107.590941] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6109.168144] Key type lgssc registered [ 6109.735714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6112.844422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6112.850556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.944983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6113.953653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6114.022334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.617928] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6118.345092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.384545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6118.396140] Lustre: 386924:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6118.494585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.018610] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6122.941568] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6124.840224] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6127.587402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.589327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6127.595767] Lustre: Skipped 1 previous similar message [ 6132.705207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6132.706875] Lustre: Skipped 2 previous similar messages [ 6137.826098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6137.830476] Lustre: Skipped 1 previous similar message [ 6140.384181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6140.464563] Lustre: server umount lustre-MDT0000 complete [ 6141.939284] LustreError: 386915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783355991 with bad export cookie 17281226746632442247 [ 6141.944463] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.947113] LustreError: 386915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6142.067891] Lustre: server umount lustre-MDT0001 complete [ 6145.906185] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:39:54 (1783355994) [ 6148.852424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.058035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.366643] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6154.209320] LustreError: 389432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.213783] LustreError: 389432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6154.434616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.547721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.023738] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6158.863036] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6160.984706] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6164.004555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.102236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6164.206154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.642885] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6169.491598] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6178.924605] Lustre: Permanently deactivating lustre-OST0000 [ 6178.931177] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6179.644902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6181.205394] LustreError: 392272:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6181.213695] LustreError: 392272:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6181.219331] LustreError: 392272:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6182.006803] LustreError: 392321:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6182.011940] LustreError: 392321:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6189.027631] Lustre: server umount lustre-OST0000 complete [ 6190.131990] LustreError: 392570:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6190.137444] LustreError: 392570:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6191.586220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.587093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6191.594506] Lustre: Skipped 1 previous similar message [ 6191.602447] Lustre: Skipped 1 previous similar message [ 6196.705055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.707400] Lustre: Skipped 1 previous similar message [ 6205.408131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6205.471069] Lustre: server umount lustre-MDT0000 complete [ 6206.686102] LustreError: 389411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356056 with bad export cookie 17281226746632442814 [ 6206.688506] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.691192] LustreError: 389411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6216.231974] Lustre: server umount lustre-MDT0001 complete [ 6220.121586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.765907] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6222.788035] LustreError: 393728:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6225.426063] Lustre: 394066:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.155@tcp [ 6226.427611] Lustre: server umount MGS complete [ 6230.991766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.183582] LustreError: 394788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.188772] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6231.194262] Lustre: Skipped 1 previous similar message [ 6231.219226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.708834] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6236.526358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.640770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6236.643158] Lustre: Skipped 2 previous similar messages [ 6236.655146] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6238.222322] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6241.229345] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6243.353226] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6244.215997] Lustre: Permanently reactivating lustre-OST0000 [ 6244.226457] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6244.231071] Lustre: Skipped 2 previous similar messages [ 6267.360159] LustreError: 396471:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6270.846032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.003831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.007549] Lustre: Skipped 1 previous similar message [ 6273.455681] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6276.336676] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6291.424894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.428068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.432987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.454303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6294.185187] Lustre: server umount lustre-OST0000 complete [ 6297.570727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.580132] Lustre: Skipped 2 previous similar messages [ 6310.368133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6310.462095] Lustre: server umount lustre-MDT0000 complete [ 6312.047163] LustreError: 394769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356161 with bad export cookie 17281226746632443836 [ 6312.054542] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.058269] LustreError: 394769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6312.218932] Lustre: server umount lustre-MDT0001 complete [ 6316.880357] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6317.888489] Key type lgssc unregistered [ 6318.036873] LNet: 399320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6318.042651] LNetError: 399320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6318.054812] LNet: Removed LNI 192.168.204.155@tcp [ 6318.540759] Key type .llcrypt unregistered [ 6318.542946] Key type ._llcrypt unregistered [ 6326.662465] Key type ._llcrypt registered [ 6326.664103] Key type .llcrypt registered [ 6326.700395] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 6333.079165] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6333.418555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6333.434874] alg: No test for adler32 (adler32-zlib) [ 6334.279276] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6334.361626] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6335.944193] Key type lgssc registered [ 6336.472472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6352.169363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6352.175048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.270927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6353.284945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6353.342473] Lustre: lustre-MDT0000: new disk, initializing [ 6353.372260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.380888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6355.122621] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6360.034715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.094941] Lustre: 402449:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6360.130528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6360.135113] Lustre: Skipped 1 previous similar message [ 6360.180531] Lustre: lustre-MDT0001: new disk, initializing [ 6360.224642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6360.242759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6360.248415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6361.991107] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6365.760695] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6368.016334] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.921321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.037230] Lustre: lustre-OST0000: new disk, initializing [ 6371.039888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6371.043159] Lustre: 404418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6371.079524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.501661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6373.505076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6373.538981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6373.579574] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6378.089311] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6379.733406] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6379.803185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6381.426786] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6381.523075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6384.102511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6384.102750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6384.115451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6384.119171] LustreError: Skipped 1 previous similar message [ 6388.466132] Lustre: server umount lustre-OST0000 complete [ 6391.265515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6391.270867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.279382] Lustre: Skipped 1 previous similar message [ 6391.283218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6391.287271] Lustre: Skipped 1 previous similar message [ 6394.338178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.338541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.467394] Lustre: server umount lustre-MDT0000 complete [ 6397.969936] LustreError: 403247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356247 with bad export cookie 14175068229464588157 [ 6397.975944] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.976636] LustreError: 403247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6398.111605] Lustre: server umount lustre-MDT0001 complete [ 6401.966675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6402.745973] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:44:11 (1783356251) [ 6405.696259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.939629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.687278] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6411.234208] LustreError: 406877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.245458] LustreError: 406877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6411.477450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.600885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6413.343549] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6416.051788] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6418.501532] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6420.881900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.072173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.623339] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6426.833212] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6427.607861] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6433.231917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6435.813793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.815034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.816659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.816666] Lustre: Skipped 1 previous similar message [ 6435.822584] Lustre: Skipped 1 previous similar message [ 6440.932003] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.936024] Lustre: Skipped 2 previous similar messages [ 6441.122961] Lustre: server umount lustre-OST0000 complete [ 6446.049715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.053356] Lustre: Skipped 1 previous similar message [ 6448.723843] Lustre: server umount lustre-MDT0000 complete [ 6449.861049] LustreError: 406904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356299 with bad export cookie 14175068229464589032 [ 6449.864163] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.864231] LustreError: 406904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6449.961555] Lustre: server umount lustre-MDT0001 complete [ 6454.131184] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6455.075515] Key type lgssc unregistered [ 6455.185340] LNet: 410596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6455.187744] LNetError: 410596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6455.199281] LNet: Removed LNI 192.168.204.155@tcp [ 6455.477542] Key type .llcrypt unregistered [ 6455.479411] Key type ._llcrypt unregistered [ 6464.157595] Key type ._llcrypt registered [ 6464.158675] Key type .llcrypt registered [ 6464.192329] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6464.278309] Key type .llcrypt unregistered [ 6464.279446] Key type ._llcrypt unregistered [ 6470.953939] Key type ._llcrypt registered [ 6470.955741] Key type .llcrypt registered [ 6470.997695] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6471.453848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6471.491917] alg: No test for adler32 (adler32-zlib) [ 6472.360855] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6472.452995] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6474.040118] Key type lgssc registered [ 6474.398474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6476.982858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6476.990794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.132795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.378377] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6482.035765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.163256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6484.329981] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6486.795123] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6487.271225] LustreError: 412779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.278626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6487.280318] LustreError: 412779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6489.428551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.631449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.046692] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6499.813244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.813579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.820719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.826629] Lustre: Skipped 1 previous similar message [ 6504.334356] Lustre: server umount lustre-OST0000 complete [ 6509.028350] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6509.035466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.043841] Lustre: Skipped 1 previous similar message [ 6509.047784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.049106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.049694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.838220] Lustre: server umount lustre-MDT0000 complete [ 6513.154410] LustreError: 412766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356363 with bad export cookie 12488973709102453487 [ 6513.158668] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.159458] LustreError: 412766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6513.289270] Lustre: server umount lustre-MDT0001 complete [ 6517.216493] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6518.418333] Key type lgssc unregistered [ 6518.559507] LNet: 415929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6518.564420] LNetError: 415929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6518.575621] LNet: Removed LNI 192.168.204.155@tcp [ 6519.020579] Key type .llcrypt unregistered [ 6519.022347] Key type ._llcrypt unregistered [ 6528.572416] Key type ._llcrypt registered [ 6528.573771] Key type .llcrypt registered [ 6528.616971] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6528.959428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6529.013988] alg: No test for adler32 (adler32-zlib) [ 6529.855697] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6529.941976] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6531.512113] Key type lgssc registered [ 6531.848923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6534.903343] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6534.911857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.368550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6555.488424] LustreError: 417295:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 6582.112399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.425342] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6586.904310] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6589.442880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.656720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.694904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6590.977521] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6594.576238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.696847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.224678] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6599.420879] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6601.363930] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6605.282362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.282516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.287354] LustreError: Skipped 1 previous similar message [ 6605.294252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6605.299216] Lustre: Skipped 1 previous similar message [ 6607.332925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.336532] Lustre: Skipped 1 previous similar message [ 6609.952206] Lustre: server umount lustre-OST0000 complete [ 6612.450759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.458608] Lustre: Skipped 1 previous similar message [ 6612.462448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.466244] Lustre: Skipped 1 previous similar message [ 6617.393108] Lustre: server umount lustre-MDT0000 complete [ 6617.569650] LustreError: 418335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.578914] LustreError: 418335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6618.988106] LustreError: 417303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356468 with bad export cookie 12744862793633571869 [ 6618.994529] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.996643] LustreError: 417303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6619.158390] Lustre: server umount lustre-MDT0001 complete [ 6623.537236] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6624.681795] Key type lgssc unregistered [ 6624.830589] LNet: 420863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.835339] LNetError: 420863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6624.853475] LNet: Removed LNI 192.168.204.155@tcp [ 6625.332663] Key type .llcrypt unregistered [ 6625.334931] Key type ._llcrypt unregistered [ 6636.059332] Key type ._llcrypt registered [ 6636.061181] Key type .llcrypt registered [ 6636.118086] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6636.468200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6636.509024] alg: No test for adler32 (adler32-zlib) [ 6637.370188] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6637.460539] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6639.040102] Key type lgssc registered [ 6639.395761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6643.011550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6643.021687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.180255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.521351] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6649.471899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.691770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6651.252922] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6654.395256] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6656.288002] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.110478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.237211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.522414] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6664.695785] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6669.486227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6674.916442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.919670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.926303] Lustre: Skipped 2 previous similar messages [ 6674.933592] Lustre: Skipped 2 previous similar messages [ 6677.358564] Lustre: server umount lustre-MDT0000 complete [ 6678.415717] LustreError: 422994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.55@tcp arrived at 1783356528 with bad export cookie 945019820936300897 [ 6678.423919] LustreError: 422994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6679.370346] LustreError: 422239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356529 with bad export cookie 945019820936300232 [ 6679.378380] LustreError: 422239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6679.379761] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.033872] LustreError: 422258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.034547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.044394] LustreError: 422258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6680.053518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6680.062751] Lustre: Skipped 1 previous similar message [ 6684.065136] LustreError: 422276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.065447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.072818] LustreError: 422276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6685.527350] Lustre: server umount lustre-OST0000 complete [ 6688.324658] Lustre: server umount lustre-MDT0001 complete [ 6691.997773] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6693.008898] Key type lgssc unregistered [ 6693.140459] LNet: 425942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6693.142623] LNetError: 425942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6693.151374] LNet: Removed LNI 192.168.204.155@tcp [ 6693.454005] Key type .llcrypt unregistered [ 6693.455894] Key type ._llcrypt unregistered [ 6704.532997] Key type ._llcrypt registered [ 6704.534092] Key type .llcrypt registered [ 6704.575432] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6705.097925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6705.109658] alg: No test for adler32 (adler32-zlib) [ 6705.968756] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6706.066920] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6707.656243] Key type lgssc registered [ 6708.236943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6711.651840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6711.657649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.824596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.338079] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6717.932154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.076711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6719.884220] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6722.769906] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6724.636682] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6727.682979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.828088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.092198] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6733.283839] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6738.093252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6743.009503] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6743.014295] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.022555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6743.526411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.537313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6744.898318] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 6748.641508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6748.643238] Lustre: Skipped 1 previous similar message [ 6753.763929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6753.767941] Lustre: Skipped 2 previous similar messages [ 6755.296138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6755.369899] Lustre: server umount lustre-MDT0001 complete [ 6758.883766] LustreError: 427336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.884272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6758.885626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.890572] LustreError: 427336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6763.309214] Lustre: server umount lustre-OST0000 complete [ 6764.003543] LustreError: 427355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.287261] Lustre: server umount lustre-MDT0000 complete [ 6771.044323] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6772.302206] Key type lgssc unregistered [ 6772.458533] LNet: 431019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6772.463405] LNetError: 431019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6772.476398] LNet: Removed LNI 192.168.204.155@tcp [ 6772.899326] Key type .llcrypt unregistered [ 6772.901046] Key type ._llcrypt unregistered [ 6781.384919] Key type ._llcrypt registered [ 6781.386422] Key type .llcrypt registered [ 6781.426936] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 6787.727663] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6788.031543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6788.055029] alg: No test for adler32 (adler32-zlib) [ 6788.903742] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6788.997832] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6790.576178] Key type lgssc registered [ 6791.082699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6804.924709] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6811.281913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6811.291600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.401057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6812.416315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6812.468712] Lustre: lustre-MDT0000: new disk, initializing [ 6812.495505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.505429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6814.323990] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6821.801074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6821.902457] Lustre: lustre-OST0000: new disk, initializing [ 6821.904524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6821.906754] Lustre: Skipped 1 previous similar message [ 6821.909267] Lustre: 436023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6821.932406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6824.366808] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6827.008620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6827.011782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6827.019546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6829.258487] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6836.564742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.614470] Lustre: 435006:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.644576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6836.680289] Lustre: lustre-MDT0001: new disk, initializing [ 6836.724596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6836.739466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6836.743088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6838.382330] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6842.572675] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6842.634133] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6844.121297] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6846.294762] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6851.045479] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6851.050593] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.059795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6851.063887] Lustre: Skipped 1 previous similar message [ 6852.943532] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 6854.764966] Lustre: server umount lustre-MDT0001 complete [ 6861.284526] LustreError: 435040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.294468] LustreError: 435040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6866.403121] LustreError: 435010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.410926] LustreError: 435010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6867.937548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6867.943403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.953125] Lustre: Skipped 1 previous similar message [ 6867.958243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6871.521614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6871.522102] LustreError: 435016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.641303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.641912] LustreError: 438340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6881.764315] LustreError: 435015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.272141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6882.352875] Lustre: server umount lustre-OST0000 complete [ 6890.395462] Lustre: server umount lustre-MDT0000 complete [ 6895.993775] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6897.196639] Key type lgssc unregistered [ 6897.343586] LNet: 439781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6897.347919] LNetError: 439781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6897.361582] LNet: Removed LNI 192.168.204.155@tcp [ 6897.753370] Key type .llcrypt unregistered [ 6897.755443] Key type ._llcrypt unregistered [ 6914.438546] Key type ._llcrypt registered [ 6914.439647] Key type .llcrypt registered [ 6914.485896] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6914.632527] Key type .llcrypt unregistered [ 6914.634791] Key type ._llcrypt unregistered [ 6922.591256] Key type ._llcrypt registered [ 6922.593046] Key type .llcrypt registered [ 6922.652711] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6922.995949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6923.045655] alg: No test for adler32 (adler32-zlib) [ 6923.906642] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 6924.007744] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6925.600211] Key type lgssc registered [ 6926.172850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6938.756173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6938.761497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.835813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6939.845112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6939.879379] Lustre: lustre-MDT0000: new disk, initializing [ 6939.897925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.903223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6941.098425] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6945.163159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.199937] Lustre: 443133:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6945.220568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6945.222411] Lustre: Skipped 1 previous similar message [ 6945.257741] Lustre: lustre-MDT0001: new disk, initializing [ 6945.286937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6945.297518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6945.300629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6946.421484] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6949.887797] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6951.428975] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6953.396408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.487717] Lustre: lustre-OST0000: new disk, initializing [ 6953.489356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6953.491776] Lustre: 445102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6953.516647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.159080] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6955.892443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6955.896237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6955.921096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6958.405889] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6960.608719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6960.611229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.615323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6961.122745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6961.122767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.124634] Lustre: Skipped 1 previous similar message [ 6961.128189] Lustre: Skipped 1 previous similar message [ 6966.241929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6966.243924] Lustre: Skipped 2 previous similar messages [ 6966.613548] Lustre: server umount lustre-MDT0000 complete [ 6967.692060] LustreError: 443123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356817 with bad export cookie 15632132621440089677 [ 6967.694743] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6967.695475] LustreError: 443123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6967.795237] Lustre: server umount lustre-MDT0001 complete [ 6979.091684] Lustre: server umount lustre-OST0000 complete [ 6981.968421] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:53:50 (1783356830) [ 6984.141071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.379512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.225327] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6989.793873] LustreError: 447129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.804939] LustreError: 447129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6990.339399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.510223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.062728] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6994.970484] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6997.474086] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7013.542654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.656714] Lustre: Found index 0 for lustre-OST0000, updating log [ 7013.764217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.104217] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7018.635888] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7030.189818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7036.385352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.390383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.398349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.770679] Lustre: server umount lustre-OST0000 complete [ 7054.766697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.767358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7054.770554] Lustre: Skipped 2 previous similar messages [ 7054.773811] Lustre: Skipped 4 previous similar messages [ 7064.033146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7064.035240] Lustre: Skipped 3 previous similar messages [ 7069.152069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7069.215972] Lustre: server umount lustre-MDT0000 complete [ 7070.177714] LustreError: 447147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.187416] LustreError: 447147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7070.534834] LustreError: 447110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783356920 with bad export cookie 15632132621440091056 [ 7070.540697] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.542428] LustreError: 447110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7070.684722] Lustre: server umount lustre-MDT0001 complete [ 7073.504432] LNet: 1 local NIs in recovery (showing 1): 192.168.204.155@tcp [ 7074.212109] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:55:23 (1783356923) [ 7655.870476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.100054] LustreError: 453565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.136777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.851590] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7661.552597] LustreError: 453566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.631164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.000788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7667.922169] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7673.086163] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7675.931497] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7691.810798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7691.901189] Lustre: Found index 0 for lustre-OST0000, updating log [ 7691.989759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7693.840820] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7696.116669] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7696.790845] Lustre: server umount lustre-OST0000 complete [ 7699.936959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7699.939502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.943101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7699.944962] Lustre: Skipped 2 previous similar messages [ 7703.009100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.009635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.155533] Lustre: server umount lustre-MDT0000 complete [ 7705.412241] LustreError: 453546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783357555 with bad export cookie 15632132621440092127 [ 7705.414557] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7705.416628] LustreError: 453546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7705.527203] Lustre: server umount lustre-MDT0001 complete [ 7713.504231] LNet: 1 local NIs in recovery (showing 1): 192.168.204.155@tcp [ 7722.878786] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 7725.890765] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 7737.530975] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:06:26 (1783357586) [ 7743.044718] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 7744.001209] Key type lgssc unregistered [ 7754.400336] LNet: 460207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7754.403796] LNetError: 460207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.966360] LNet: Removed LNI 192.168.204.155@tcp [ 7821.266711] Key type .llcrypt unregistered [ 7821.267926] Key type ._llcrypt unregistered [ 7828.254865] Key type ._llcrypt registered [ 7828.256423] Key type .llcrypt registered [ 7828.301882] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 7828.640140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7828.682751] alg: No test for adler32 (adler32-zlib) [ 7829.538734] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 7829.627979] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 7831.208198] Key type lgssc registered [ 7831.652139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7833.463234] LNet: Added LNI 192.168.204.155@tcp100 [8/256/0/180] [ 7833.847255] LNet: Added LNI 192.168.204.155@tcp101 [8/256/0/180] [ 7834.203231] LNet: Added LNI 192.168.204.155@tcp102 [8/256/0/180] [ 7834.533055] LNet: Added LNI 192.168.204.155@tcp103 [8/256/0/180] [ 7834.845278] LNet: Added LNI 192.168.204.155@tcp104 [8/256/0/180] [ 7835.162506] LNet: Added LNI 192.168.204.155@tcp105 [8/256/0/180] [ 7835.502251] LNet: Added LNI 192.168.204.155@tcp106 [8/256/0/180] [ 7835.846933] LNet: Added LNI 192.168.204.155@tcp107 [8/256/0/180] [ 7836.170760] LNet: Added LNI 192.168.204.155@tcp108 [8/256/0/180] [ 7836.500254] LNet: Added LNI 192.168.204.155@tcp109 [8/256/0/180] [ 7836.817366] LNet: Added LNI 192.168.204.155@tcp110 [8/256/0/180] [ 7837.146828] LNet: Added LNI 192.168.204.155@tcp111 [8/256/0/180] [ 7837.471282] LNet: Added LNI 192.168.204.155@tcp112 [8/256/0/180] [ 7837.788454] LNet: Added LNI 192.168.204.155@tcp113 [8/256/0/180] [ 7838.104236] LNet: Added LNI 192.168.204.155@tcp114 [8/256/0/180] [ 7838.404416] LNet: Added LNI 192.168.204.155@tcp115 [8/256/0/180] [ 7838.722401] LNet: Added LNI 192.168.204.155@tcp116 [8/256/0/180] [ 7839.054704] LNet: Added LNI 192.168.204.155@tcp117 [8/256/0/180] [ 7839.392730] LNet: Added LNI 192.168.204.155@tcp118 [8/256/0/180] [ 7839.708504] LNet: Added LNI 192.168.204.155@tcp119 [8/256/0/180] [ 7840.030852] LNet: Added LNI 192.168.204.155@tcp120 [8/256/0/180] [ 7840.347995] LNet: Added LNI 192.168.204.155@tcp121 [8/256/0/180] [ 7840.653806] LNet: Added LNI 192.168.204.155@tcp122 [8/256/0/180] [ 7840.958484] LNet: Added LNI 192.168.204.155@tcp123 [8/256/0/180] [ 7841.260597] LNet: Added LNI 192.168.204.155@tcp124 [8/256/0/180] [ 7841.573217] LNet: Added LNI 192.168.204.155@tcp125 [8/256/0/180] [ 7841.885185] LNet: Added LNI 192.168.204.155@tcp126 [8/256/0/180] [ 7842.201952] LNet: Added LNI 192.168.204.155@tcp127 [8/256/0/180] [ 7842.507910] LNet: Added LNI 192.168.204.155@tcp128 [8/256/0/180] [ 7842.818092] LNet: Added LNI 192.168.204.155@tcp129 [8/256/0/180] [ 7843.132558] LNet: Added LNI 192.168.204.155@tcp130 [8/256/0/180] [ 7843.423086] LNet: Added LNI 192.168.204.155@tcp131 [8/256/0/180] [ 7843.753639] LNet: Added LNI 192.168.204.155@tcp132 [8/256/0/180] [ 7844.069193] LNet: Added LNI 192.168.204.155@tcp133 [8/256/0/180] [ 7844.391300] LNet: Added LNI 192.168.204.155@tcp134 [8/256/0/180] [ 7844.721177] LNet: Added LNI 192.168.204.155@tcp135 [8/256/0/180] [ 7847.254660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7847.262753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.304983] Lustre: 463205:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.155@tcp: can't add NID 192.168.204.155@tcp130: rc = -75 [ 7848.364863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7848.377195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7848.417183] Lustre: lustre-MDT0000: new disk, initializing [ 7848.451806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.459460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7850.000944] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7854.398183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.450871] Lustre: 463224:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7854.484256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7854.487141] Lustre: Skipped 1 previous similar message [ 7854.524559] Lustre: lustre-MDT0001: new disk, initializing [ 7854.536749] Lustre: 464023:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.155@tcp: can't add NID 192.168.204.155@tcp130: rc = -75 [ 7854.540362] Lustre: 464023:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7854.581538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7854.593663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7854.599363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7855.996667] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7859.900579] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7861.849556] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7864.315621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.426501] Lustre: lustre-OST0000: new disk, initializing [ 7864.428925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7864.432427] Lustre: 465195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7864.468719] Lustre: 465200:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.155@tcp: can't add NID 192.168.204.155@tcp130: rc = -75 [ 7864.471953] Lustre: 465200:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7864.492159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.610670] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7870.539648] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7873.013543] Lustre: 466112:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.155@tcp: can't add NID 192.168.204.155@tcp130: rc = -75 [ 7873.017213] Lustre: 466112:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7873.037042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7873.041642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7873.100252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7878.113802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7878.115436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.117508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7878.121597] Lustre: Skipped 1 previous similar message [ 7879.328441] Lustre: server umount lustre-OST0000 complete [ 7883.234246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.235076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.238093] Lustre: Skipped 1 previous similar message [ 7883.241154] Lustre: Skipped 1 previous similar message [ 7886.814920] Lustre: server umount lustre-MDT0000 complete [ 7888.168254] LustreError: 464021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783357738 with bad export cookie 11288029021173337966 [ 7888.170841] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.173892] LustreError: 464021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7888.305602] Lustre: server umount lustre-MDT0001 complete [ 7892.253076] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 7893.298170] Key type lgssc unregistered [ 7893.464364] LNet: 467105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7893.468897] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7893.472398] LNet: Removed LNI 192.168.204.155@tcp [ 7894.496122] LNet: Removed LNI 192.168.204.155@tcp100 [ 7894.497704] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.500436] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7894.503624] LNet: Removed LNI 192.168.204.155@tcp101 [ 7895.520141] LNet: Removed LNI 192.168.204.155@tcp102 [ 7895.522129] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7895.525878] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7895.529067] LNet: Removed LNI 192.168.204.155@tcp103 [ 7896.544588] LNet: Removed LNI 192.168.204.155@tcp104 [ 7897.568177] LNet: Removed LNI 192.168.204.155@tcp105 [ 7897.570640] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7897.573342] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7897.577390] LNet: Removed LNI 192.168.204.155@tcp106 [ 7898.592139] LNet: Removed LNI 192.168.204.155@tcp107 [ 7899.616140] LNet: Removed LNI 192.168.204.155@tcp108 [ 7900.640127] LNet: Removed LNI 192.168.204.155@tcp109 [ 7901.664150] LNet: Removed LNI 192.168.204.155@tcp110 [ 7901.666300] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7901.670019] LNetError: 467105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7901.673878] LNet: Removed LNI 192.168.204.155@tcp111 [ 7901.675943] LNet: Removed LNI 192.168.204.155@tcp112 [ 7901.677737] LNet: Removed LNI 192.168.204.155@tcp113 [ 7901.679770] LNet: Removed LNI 192.168.204.155@tcp114 [ 7901.681713] LNet: Removed LNI 192.168.204.155@tcp115 [ 7901.683687] LNet: Removed LNI 192.168.204.155@tcp116 [ 7901.685897] LNet: Removed LNI 192.168.204.155@tcp117 [ 7901.687825] LNet: Removed LNI 192.168.204.155@tcp118 [ 7901.689848] LNet: Removed LNI 192.168.204.155@tcp119 [ 7901.691798] LNet: Removed LNI 192.168.204.155@tcp120 [ 7901.693827] LNet: Removed LNI 192.168.204.155@tcp121 [ 7901.695673] LNet: Removed LNI 192.168.204.155@tcp122 [ 7901.697697] LNet: Removed LNI 192.168.204.155@tcp123 [ 7901.699760] LNet: Removed LNI 192.168.204.155@tcp124 [ 7901.701872] LNet: Removed LNI 192.168.204.155@tcp125 [ 7901.703769] LNet: Removed LNI 192.168.204.155@tcp126 [ 7901.705733] LNet: Removed LNI 192.168.204.155@tcp127 [ 7901.707730] LNet: Removed LNI 192.168.204.155@tcp128 [ 7901.709236] LNet: Removed LNI 192.168.204.155@tcp129 [ 7901.710543] LNet: Removed LNI 192.168.204.155@tcp130 [ 7901.711893] LNet: Removed LNI 192.168.204.155@tcp131 [ 7901.713345] LNet: Removed LNI 192.168.204.155@tcp132 [ 7901.715067] LNet: Removed LNI 192.168.204.155@tcp133 [ 7901.716449] LNet: Removed LNI 192.168.204.155@tcp134 [ 7901.724398] LNet: Removed LNI 192.168.204.155@tcp135 [ 7902.029042] Key type .llcrypt unregistered [ 7902.030235] Key type ._llcrypt unregistered [ 7909.828528] Key type ._llcrypt registered [ 7909.829566] Key type .llcrypt registered [ 7909.864809] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 7916.452989] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 7916.810382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7916.844462] alg: No test for adler32 (adler32-zlib) [ 7917.714554] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 7917.819122] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 7919.408170] Key type lgssc registered [ 7919.819983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7932.400403] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:09:41 (1783357781) [ 7939.161244] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 7940.334696] Key type lgssc unregistered [ 7940.474380] LNet: 471160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.478796] LNetError: 471160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.494821] LNet: Removed LNI 192.168.204.155@tcp [ 7940.883540] Key type .llcrypt unregistered [ 7940.885486] Key type ._llcrypt unregistered [ 7948.479476] Key type ._llcrypt registered [ 7948.481158] Key type .llcrypt registered [ 7948.526220] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 7948.914561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7948.951725] alg: No test for adler32 (adler32-zlib) [ 7949.818424] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 7949.915440] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 7951.504178] Key type lgssc registered [ 7951.908595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.313882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7955.321368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.405975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7956.416041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7956.449088] Lustre: lustre-MDT0000: new disk, initializing [ 7956.468889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7956.474844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7957.910446] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7962.470332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.521144] Lustre: 472350:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7962.545886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7962.548493] Lustre: Skipped 1 previous similar message [ 7962.585052] Lustre: lustre-MDT0001: new disk, initializing [ 7962.617613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7962.630143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7962.634843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7964.100048] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7967.994668] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7969.906771] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7972.186905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7972.298659] Lustre: lustre-OST0000: new disk, initializing [ 7972.301502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7972.305852] Lustre: 474320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7972.338616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7974.372310] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 7978.015183] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7979.164484] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing lnet_if_list [ 7980.764553] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7980.779698] LNet: Added LNI 192.168.204.155@tcp10 [8/256/0/180] [ 7981.049718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7981.055396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7981.089674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7982.447346] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7982.459754] LNet: Added LNI 192.168.204.155@tcp11 [8/256/0/180] [ 7983.789624] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7983.800744] LNet: Added LNI 192.168.204.155@tcp12 [8/256/0/180] [ 7985.151034] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7985.161903] LNet: Added LNI 192.168.204.155@tcp13 [8/256/0/180] [ 7986.457268] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7986.470140] LNet: Added LNI 192.168.204.155@tcp14 [8/256/0/180] [ 7987.810705] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7987.822857] LNet: Added LNI 192.168.204.155@tcp15 [8/256/0/180] [ 7989.154301] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7989.166355] LNet: Added LNI 192.168.204.155@tcp16 [8/256/0/180] [ 7990.588399] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7990.601089] LNet: Added LNI 192.168.204.155@tcp17 [8/256/0/180] [ 7992.094606] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7992.105143] LNet: Added LNI 192.168.204.155@tcp18 [8/256/0/180] [ 7993.697131] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7993.708466] LNet: Added LNI 192.168.204.155@tcp19 [8/256/0/180] [ 7995.326291] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7995.339982] LNet: Added LNI 192.168.204.155@tcp20 [8/256/0/180] [ 7997.015023] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7997.027695] LNet: Added LNI 192.168.204.155@tcp21 [8/256/0/180] [ 7998.748274] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7998.762510] LNet: Added LNI 192.168.204.155@tcp22 [8/256/0/180] [ 8000.497078] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8000.506657] LNet: Added LNI 192.168.204.155@tcp23 [8/256/0/180] [ 8002.300831] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8002.312931] LNet: Added LNI 192.168.204.155@tcp24 [8/256/0/180] [ 8004.131416] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8004.145365] LNet: Added LNI 192.168.204.155@tcp25 [8/256/0/180] [ 8005.984571] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8005.996262] LNet: Added LNI 192.168.204.155@tcp26 [8/256/0/180] [ 8007.945848] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8007.959723] LNet: Added LNI 192.168.204.155@tcp27 [8/256/0/180] [ 8009.887498] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8009.900193] LNet: Added LNI 192.168.204.155@tcp28 [8/256/0/180] [ 8011.868632] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8011.883323] LNet: Added LNI 192.168.204.155@tcp29 [8/256/0/180] [ 8014.045593] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8014.059743] LNet: Added LNI 192.168.204.155@tcp30 [8/256/0/180] [ 8016.154159] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8016.168273] LNet: Added LNI 192.168.204.155@tcp31 [8/256/0/180] [ 8018.302238] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8018.313849] LNet: Added LNI 192.168.204.155@tcp32 [8/256/0/180] [ 8020.527393] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8020.543353] LNet: Added LNI 192.168.204.155@tcp33 [8/256/0/180] [ 8022.758448] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8022.772437] LNet: Added LNI 192.168.204.155@tcp34 [8/256/0/180] [ 8024.979918] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8024.995347] LNet: Added LNI 192.168.204.155@tcp35 [8/256/0/180] [ 8027.418844] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8027.435086] LNet: Added LNI 192.168.204.155@tcp36 [8/256/0/180] [ 8029.909572] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8029.922489] LNet: Added LNI 192.168.204.155@tcp37 [8/256/0/180] [ 8032.281538] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8032.291450] LNet: Added LNI 192.168.204.155@tcp38 [8/256/0/180] [ 8034.618152] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8034.632126] LNet: Added LNI 192.168.204.155@tcp39 [8/256/0/180] [ 8036.923490] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8036.935327] LNet: Added LNI 192.168.204.155@tcp40 [8/256/0/180] [ 8039.316682] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8039.328774] LNet: Added LNI 192.168.204.155@tcp41 [8/256/0/180] [ 8041.915234] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8041.928943] LNet: Added LNI 192.168.204.155@tcp42 [8/256/0/180] [ 8044.522425] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8044.535055] LNet: Added LNI 192.168.204.155@tcp43 [8/256/0/180] [ 8047.274390] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8047.287381] LNet: Added LNI 192.168.204.155@tcp44 [8/256/0/180] [ 8049.922860] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8049.938595] LNet: Added LNI 192.168.204.155@tcp45 [8/256/0/180] [ 8052.680141] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8052.692572] LNet: Added LNI 192.168.204.155@tcp46 [8/256/0/180] [ 8055.482346] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8055.493180] LNet: Added LNI 192.168.204.155@tcp47 [8/256/0/180] [ 8058.249335] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8058.263383] LNet: Added LNI 192.168.204.155@tcp48 [8/256/0/180] [ 8061.180577] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8061.194249] LNet: Added LNI 192.168.204.155@tcp49 [8/256/0/180] [ 8064.110830] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8064.124184] LNet: Added LNI 192.168.204.155@tcp50 [8/256/0/180] [ 8073.187234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.191866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8073.192108] Lustre: Skipped 2 previous similar messages [ 8073.196978] Lustre: Skipped 2 previous similar messages [ 8078.307207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8078.309603] Lustre: Skipped 2 previous similar messages [ 8078.642331] Lustre: server umount lustre-MDT0000 complete [ 8080.132430] LustreError: 472342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783357929 with bad export cookie 176766204989536354 [ 8080.136982] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8080.137921] LustreError: 472342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8080.277569] Lustre: server umount lustre-MDT0001 complete [ 8088.260502] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8091.300180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8099.808120] Lustre: 471625:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783357933/real 1783357933] req@0000000066ffe81b x1869986187527552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783357949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8099.817371] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.600153] Lustre: 471626:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783357935/real 1783357935] req@00000000c8b3995b x1869986187527680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783357951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8104.928134] Lustre: 471624:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783357938/real 1783357938] req@00000000043c337c x1869986187527808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783357954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8105.952333] LustreError: 471622:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a157e22d x1869986187528576/t0(0) o250->MGC192.168.204.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8106.049895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.104118] Lustre: 471623:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783357940/real 1783357940] req@000000002dbbc5b7 x1869986187528064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783357956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8107.343801] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8110.068499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.112883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8110.115080] Lustre: Skipped 1 previous similar message [ 8110.186420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8111.200099] Lustre: 471625:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783357945/real 1783357945] req@000000005187c0f4 x1869986187528192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783357961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8111.558767] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8113.914964] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8115.674491] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8115.685844] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8125.408920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8125.411021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.414801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8128.850507] Lustre: server umount lustre-OST0000 complete [ 8130.529194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8130.532717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8130.538274] Lustre: Skipped 1 previous similar message [ 8130.542249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.544983] Lustre: Skipped 1 previous similar message [ 8136.271657] Lustre: server umount lustre-MDT0000 complete [ 8137.442169] LustreError: 473148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783357987 with bad export cookie 176766204989538426 [ 8137.444349] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.445789] LustreError: 473148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8137.550036] Lustre: server umount lustre-MDT0001 complete [ 8141.004468] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 8141.935711] Key type lgssc unregistered [ 8142.091814] LNet: 501280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8142.094840] LNetError: 501280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8142.096752] LNet: Removed LNI 192.168.204.155@tcp [ 8142.098399] LNet: Removed LNI 192.168.204.155@tcp10 [ 8142.100078] LNet: Removed LNI 192.168.204.155@tcp11 [ 8142.101170] LNet: Removed LNI 192.168.204.155@tcp12 [ 8142.102569] LNet: Removed LNI 192.168.204.155@tcp13 [ 8142.103868] LNet: Removed LNI 192.168.204.155@tcp14 [ 8142.105035] LNet: Removed LNI 192.168.204.155@tcp15 [ 8142.105972] LNet: Removed LNI 192.168.204.155@tcp16 [ 8142.107067] LNet: Removed LNI 192.168.204.155@tcp17 [ 8142.108095] LNet: Removed LNI 192.168.204.155@tcp18 [ 8142.109114] LNet: Removed LNI 192.168.204.155@tcp19 [ 8142.110135] LNet: Removed LNI 192.168.204.155@tcp20 [ 8142.111237] LNet: Removed LNI 192.168.204.155@tcp21 [ 8142.112590] LNet: Removed LNI 192.168.204.155@tcp22 [ 8142.113628] LNet: Removed LNI 192.168.204.155@tcp23 [ 8142.114774] LNet: Removed LNI 192.168.204.155@tcp24 [ 8142.116327] LNet: Removed LNI 192.168.204.155@tcp25 [ 8142.118175] LNet: Removed LNI 192.168.204.155@tcp26 [ 8142.119193] LNet: Removed LNI 192.168.204.155@tcp27 [ 8142.120201] LNet: Removed LNI 192.168.204.155@tcp28 [ 8142.121324] LNet: Removed LNI 192.168.204.155@tcp29 [ 8142.122553] LNet: Removed LNI 192.168.204.155@tcp30 [ 8142.124192] LNet: Removed LNI 192.168.204.155@tcp31 [ 8142.125700] LNet: Removed LNI 192.168.204.155@tcp32 [ 8142.127050] LNet: Removed LNI 192.168.204.155@tcp33 [ 8142.128329] LNet: Removed LNI 192.168.204.155@tcp34 [ 8142.129922] LNet: Removed LNI 192.168.204.155@tcp35 [ 8142.131483] LNet: Removed LNI 192.168.204.155@tcp36 [ 8142.132658] LNet: Removed LNI 192.168.204.155@tcp37 [ 8142.133869] LNet: Removed LNI 192.168.204.155@tcp38 [ 8142.135055] LNet: Removed LNI 192.168.204.155@tcp39 [ 8142.136241] LNet: Removed LNI 192.168.204.155@tcp40 [ 8142.137333] LNet: Removed LNI 192.168.204.155@tcp41 [ 8142.138586] LNet: Removed LNI 192.168.204.155@tcp42 [ 8142.139890] LNet: Removed LNI 192.168.204.155@tcp43 [ 8142.141089] LNet: Removed LNI 192.168.204.155@tcp44 [ 8142.142283] LNet: Removed LNI 192.168.204.155@tcp45 [ 8142.143802] LNet: Removed LNI 192.168.204.155@tcp46 [ 8142.145699] LNet: Removed LNI 192.168.204.155@tcp47 [ 8142.147527] LNet: Removed LNI 192.168.204.155@tcp48 [ 8142.149181] LNet: Removed LNI 192.168.204.155@tcp49 [ 8142.157368] LNet: Removed LNI 192.168.204.155@tcp50 [ 8142.438833] Key type .llcrypt unregistered [ 8142.439970] Key type ._llcrypt unregistered [ 8148.667963] Key type ._llcrypt registered [ 8148.668860] Key type .llcrypt registered [ 8148.704628] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 8154.506341] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8155.041812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8155.052079] alg: No test for adler32 (adler32-zlib) [ 8155.902211] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 8155.988933] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 8157.576157] Key type lgssc registered [ 8157.930686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8168.474418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8168.478856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.572415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8169.582645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8169.621275] Lustre: lustre-MDT0000: new disk, initializing [ 8169.643792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.650945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8171.013642] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8175.153210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.191103] Lustre: 504449:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8175.213161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8175.215242] Lustre: Skipped 1 previous similar message [ 8175.246222] Lustre: lustre-MDT0001: new disk, initializing [ 8175.277373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8175.288180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8175.290808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8176.617596] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8180.203934] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8181.855125] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8184.267400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8184.363940] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8184.396332] Lustre: lustre-OST0000: new disk, initializing [ 8184.398202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8184.400716] Lustre: 506464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8184.430971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.334031] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8189.836715] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8192.175288] Lustre: server umount lustre-OST0000 complete [ 8194.546504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.547427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.550826] Lustre: Skipped 1 previous similar message [ 8199.577810] Lustre: server umount lustre-MDT0000 complete [ 8200.831204] LustreError: 504441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783358050 with bad export cookie 13662584491907069862 [ 8200.833601] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8200.835320] LustreError: 504441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8200.946144] Lustre: server umount lustre-MDT0001 complete [ 8204.355131] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 8205.327657] Key type lgssc unregistered [ 8205.457537] LNet: 508358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8205.461239] LNetError: 508358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8205.470503] LNet: Removed LNI 192.168.204.155@tcp [ 8205.780563] Key type .llcrypt unregistered [ 8205.781854] Key type ._llcrypt unregistered [ 8212.884196] Key type ._llcrypt registered [ 8212.885265] Key type .llcrypt registered [ 8212.919312] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8213.275992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8213.307298] alg: No test for adler32 (adler32-zlib) [ 8214.150261] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 8214.231820] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 8215.808135] Key type lgssc registered [ 8216.160447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8218.563445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8218.570905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.707771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.951669] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8223.590208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.700340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8224.944810] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8227.193120] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8228.961777] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.057516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.160635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8232.956367] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8233.191032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8233.194384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8235.302039] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8244.463942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8249.559311] loop0: detected capacity change from 0 to 204800000 [ 8249.620093] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8249.623779] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8249.628379] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8251.565243] loop0: detected capacity change from 0 to 204800000 [ 8251.690085] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8251.694132] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8251.698893] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8253.288295] loop0: detected capacity change from 0 to 204800000 [ 8255.054945] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8255.114614] Lustre: fs15246-OST0000: new disk, initializing [ 8255.116383] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8255.120340] Lustre: 513216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8255.141973] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.864236] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8260.486977] loop1: detected capacity change from 0 to 204800000 [ 8262.275724] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.314361] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8262.335142] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8262.380479] Lustre: fs15246-MDT0000: new disk, initializing [ 8262.428482] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.439050] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8262.486510] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8263.520788] Lustre: 514338:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000002af22927 x1869986465349248/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783358129 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8263.710339] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8271.847506] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.608218] LustreError: 514338:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8272.611969] LustreError: 514338:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8272.615345] LustreError: 514338:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8276.814330] Lustre: server umount fs15246-MDT0000 complete [ 8276.964366] LustreError: 510332:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.968542] LustreError: 510332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8279.652928] Lustre: server umount fs15246-OST0000 complete [ 8284.640813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.643103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.647252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.664732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.667188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.671739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8288.910721] Lustre: server umount lustre-OST0000 complete [ 8290.272756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8290.275935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.280453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.282282] Lustre: Skipped 2 previous similar messages [ 8292.321667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.148523] Lustre: server umount lustre-MDT0000 complete [ 8297.265924] LustreError: 509586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783358147 with bad export cookie 16389942951305485858 [ 8297.269129] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.269294] LustreError: 509586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8297.378917] Lustre: server umount lustre-MDT0001 complete [ 8300.611327] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 8301.560253] Key type lgssc unregistered [ 8301.685363] LNet: 517433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8301.688683] LNetError: 517433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8301.695331] LNet: Removed LNI 192.168.204.155@tcp [ 8301.995763] Key type .llcrypt unregistered [ 8301.996909] Key type ._llcrypt unregistered [ 8308.694113] Key type ._llcrypt registered [ 8308.695260] Key type .llcrypt registered [ 8308.727996] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8309.011044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8309.058062] alg: No test for adler32 (adler32-zlib) [ 8309.902419] Lustre: Lustre: Build Version: 2.17.54_84_g95d2d66 [ 8309.985890] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 8311.560111] Key type lgssc registered [ 8311.867640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8323.189642] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 8325.983529] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8336.546779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8336.552121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.626846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8337.636682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8337.669501] Lustre: lustre-MDT0000: new disk, initializing [ 8337.689372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8337.695310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8338.929707] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8342.708607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.746048] Lustre: 521869:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8342.768826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8342.770815] Lustre: Skipped 1 previous similar message [ 8342.802760] Lustre: lustre-MDT0001: new disk, initializing [ 8342.830629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.842396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8342.847613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8344.052544] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8347.571586] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8349.273801] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8351.465535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.566491] Lustre: lustre-OST0000: new disk, initializing [ 8351.568382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8351.570963] Lustre: 523834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8351.597694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.370366] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 8354.102061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8354.104863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8354.131037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8356.853267] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8358.116198] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8358.179893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8359.423956] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8359.491722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8362.190401] Lustre: server umount lustre-OST0000 complete [ 8363.488785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8363.491590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.495552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.513149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.513416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.633114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8369.635378] Lustre: Skipped 2 previous similar messages [ 8373.750480] Lustre: server umount lustre-MDT0000 complete [ 8374.964271] LustreError: 521860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783358224 with bad export cookie 17234567995214780502 [ 8374.967909] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8374.969347] LustreError: 521860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8384.544873] Lustre: server umount lustre-MDT0001 complete [ 8392.148165] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 8395.024310] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 8404.409995] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8086 sec ========== 13:17:33 (1783358253) [ 8404.901761] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:17:33 (1783358253) === [ 8405.918237] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:17:34 (1783358254) === [ 8418.314333] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 8419.339992] Key type lgssc unregistered [ 8419.475395] LNet: 530463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8419.477888] LNetError: 530463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8419.487320] LNet: Removed LNI 192.168.204.155@tcp [ 8419.811054] Key type .llcrypt unregistered [ 8419.811960] Key type ._llcrypt unregistered