[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 539474687 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003210] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007855] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013047] *** VALIDATE selinux *** [ 0.020153] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024805] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026045] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028072] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] *** VALIDATE tmpfs *** [ 0.030453] *** VALIDATE proc *** [ 0.031249] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040823] debug: unmapping init [mem 0xffffffff8ce59000-0xffffffff8ce60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.050000] ... event mask: 000000070000000f [ 0.051306] rcu: Hierarchical SRCU implementation. [ 0.053215] smp: Bringing up secondary CPUs ... [ 0.054451] x86: Booting SMP configuration: [ 0.055031] .... node #0, CPUs: #1 #2 #3 [ 0.058351] smp: Brought up 1 node, 4 CPUs [ 0.059860] smpboot: Max logical packages: 1 [ 0.060010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143195] node 0 deferred pages initialised in 80ms [ 0.146096] devtmpfs: initialized [ 0.147233] x86/mm: Memory block size: 128MB [ 0.149939] gcov: version magic: 0x41383552 [ 0.151367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155268] pinctrl core: initialized pinctrl subsystem [ 0.156200] [ 0.156895] ************************************************************* [ 0.157012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158014] ** ** [ 0.159010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161012] ** This means that this kernel is built to expose internal ** [ 0.162014] ** IOMMU data structures, which may compromise security on ** [ 0.163011] ** your system. ** [ 0.164014] ** ** [ 0.165014] ** If you see this message and you are not debugging the ** [ 0.166016] ** kernel, report this immediately to your vendor! ** [ 0.167015] ** ** [ 0.168016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169019] ************************************************************* [ 0.170725] NET: Registered protocol family 16 [ 0.171448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174335] cpuidle: using governor menu [ 0.176174] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178593] PCI: Using configuration type 1 for base access [ 0.180112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192082] cryptd: max_cpu_qlen set to 1000 [ 0.195241] ACPI: Added _OSI(Module Device) [ 0.196012] ACPI: Added _OSI(Processor Device) [ 0.197254] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202309] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205096] ACPI: Interpreter enabled [ 0.206048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207010] ACPI: Using IOAPIC for interrupt routing [ 0.208108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219569] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.222079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224325] acpiphp: Slot [2] registered [ 0.225097] acpiphp: Slot [5] registered [ 0.226093] acpiphp: Slot [6] registered [ 0.227122] acpiphp: Slot [7] registered [ 0.228101] acpiphp: Slot [8] registered [ 0.229092] acpiphp: Slot [9] registered [ 0.230098] acpiphp: Slot [10] registered [ 0.231128] acpiphp: Slot [3] registered [ 0.232082] acpiphp: Slot [4] registered [ 0.233119] acpiphp: Slot [11] registered [ 0.234081] acpiphp: Slot [12] registered [ 0.235086] acpiphp: Slot [13] registered [ 0.236101] acpiphp: Slot [14] registered [ 0.237089] acpiphp: Slot [15] registered [ 0.238088] acpiphp: Slot [16] registered [ 0.239094] acpiphp: Slot [17] registered [ 0.240117] acpiphp: Slot [18] registered [ 0.241140] acpiphp: Slot [19] registered [ 0.242101] acpiphp: Slot [20] registered [ 0.243095] acpiphp: Slot [21] registered [ 0.244100] acpiphp: Slot [22] registered [ 0.245085] acpiphp: Slot [23] registered [ 0.246108] acpiphp: Slot [24] registered [ 0.247087] acpiphp: Slot [25] registered [ 0.248086] acpiphp: Slot [26] registered [ 0.249126] acpiphp: Slot [27] registered [ 0.250090] acpiphp: Slot [28] registered [ 0.251064] acpiphp: Slot [29] registered [ 0.252095] acpiphp: Slot [30] registered [ 0.253110] acpiphp: Slot [31] registered [ 0.254047] PCI host bridge to bus 0000:00 [ 0.255021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.257019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.258020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.259024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.262978] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.264314] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.270015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.273123] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.274017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.276015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.277606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.278851] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.279048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.280571] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.283014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.291835] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.293018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.297072] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.300017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.303016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.311016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.319126] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.335014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.348025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.367015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.377680] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.389015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.403016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.443026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.452841] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.471016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.479015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.498016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.507277] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.518024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.526024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.558372] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.565026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.570014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.588026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.598342] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.601426] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.603464] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.606418] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.609459] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.615078] iommu: Default domain type: Passthrough [ 0.616341] SCSI subsystem initialized [ 0.617149] ACPI: bus type USB registered [ 0.618245] usbcore: registered new interface driver usbfs [ 0.620086] usbcore: registered new interface driver hub [ 0.622105] usbcore: registered new device driver usb [ 0.624244] pps_core: LinuxPPS API ver. 1 registered [ 0.626012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.629064] PTP clock support registered [ 0.631142] EDAC MC: Ver: 3.0.0 [ 0.633174] PCI: Using ACPI for IRQ routing [ 0.636000] NetLabel: Initializing [ 0.637018] NetLabel: domain hash size = 128 [ 0.638016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.640112] NetLabel: unlabeled traffic allowed by default [ 0.642129] vgaarb: loaded [ 0.643243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.644010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.651044] clocksource: Switched to clocksource kvm-clock [ 0.755841] VFS: Disk quotas dquot_6.6.0 [ 0.757366] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759885] *** VALIDATE ramfs *** [ 0.761158] *** VALIDATE hugetlbfs *** [ 0.763344] pnp: PnP ACPI init [ 0.765842] pnp: PnP ACPI: found 6 devices [ 0.783288] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.785921] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.787781] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.790019] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.791835] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.794155] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.796544] NET: Registered protocol family 2 [ 0.798751] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.803426] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.806382] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.811411] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.814379] TCP: Hash tables configured (established 65536 bind 65536) [ 0.817075] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.819796] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.822202] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.824362] NET: Registered protocol family 1 [ 0.826367] RPC: Registered named UNIX socket transport module. [ 0.827887] RPC: Registered udp transport module. [ 0.829054] RPC: Registered tcp transport module. [ 0.830429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.832120] NET: Registered protocol family 44 [ 0.833225] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.834612] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836086] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.837661] PCI: CLS 0 bytes, default 64 [ 0.839424] Unpacking initramfs... [ 2.212616] debug: unmapping init [mem 0xffff8d7abcc54000-0xffff8d7abffbffff] [ 2.219311] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.221461] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225231] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.722684] Initialise system trusted keyrings [ 2.723936] Key type blacklist registered [ 2.725967] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.733947] zbud: loaded [ 2.739188] *** VALIDATE nfs *** [ 2.740194] *** VALIDATE nfs4 *** [ 2.742048] pstore: using deflate compression [ 2.746218] Platform Keyring initialized [ 2.858878] NET: Registered protocol family 38 [ 2.860937] Key type asymmetric registered [ 2.862686] Asymmetric key parser 'x509' registered [ 2.864866] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.868389] io scheduler mq-deadline registered [ 2.870388] io scheduler kyber registered [ 2.872270] io scheduler bfq registered [ 2.874462] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.877943] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.881281] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.884425] ACPI: Power Button [PWRF] [ 2.889558] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.895044] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.908521] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.915414] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.925835] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.951868] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.978954] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.982691] Non-volatile memory driver v1.3 [ 2.984356] Linux agpgart interface v0.103 [ 3.012735] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.015188] vda: detected capacity change from 0 to 76775424 [ 3.035500] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.038866] vdb: detected capacity change from 0 to 1073741824 [ 3.057675] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.061092] vdc: detected capacity change from 0 to 2621440000 [ 3.084517] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.088356] vdd: detected capacity change from 0 to 2621440000 [ 3.104604] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107921] vde: detected capacity change from 0 to 4294967296 [ 3.125334] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.128865] vdf: detected capacity change from 0 to 4294967296 [ 3.142154] libphy: Fixed MDIO Bus: probed [ 3.150238] usbcore: registered new interface driver usbserial_generic [ 3.152409] usbserial: USB Serial support registered for generic [ 3.154111] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.157620] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.159093] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.160990] mousedev: PS/2 mouse device common for all mice [ 3.163821] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.166325] rtc_cmos 00:05: RTC can wake from S4 [ 3.171208] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.173307] rtc_cmos 00:05: registered as rtc0 [ 3.177553] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.178142] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.185119] intel_pstate: CPU model not supported [ 3.188567] hid: raw HID events driver (C) Jiri Kosina [ 3.190967] usbcore: registered new interface driver usbhid [ 3.193341] usbhid: USB HID core driver [ 3.194941] drop_monitor: Initializing network drop monitor service [ 3.197682] Initializing XFRM netlink socket [ 3.199846] NET: Registered protocol family 10 [ 3.203236] Segment Routing with IPv6 [ 3.204491] NET: Registered protocol family 17 [ 3.206135] mpls_gso: MPLS GSO support [ 3.212905] RAS: Correctable Errors collector initialized. [ 3.214781] AVX version of gcm_enc/dec engaged. [ 3.216135] AES CTR mode by8 optimization enabled [ 3.293304] sched_clock: Marking stable (3293279539, 0)->(4252770924, -959491385) [ 3.296728] registered taskstats version 1 [ 3.298865] Loading compiled-in X.509 certificates [ 3.300996] zswap: loaded using pool lzo/zbud [ 3.325949] Key type big_key registered [ 3.337676] Key type encrypted registered [ 3.339495] ima: No TPM chip found, activating TPM-bypass! [ 3.341662] ima: Allocated hash algorithm: sha1 [ 3.343608] ima: No architecture policies found [ 3.345287] evm: Initialising EVM extended attributes: [ 3.346732] evm: security.selinux [ 3.347707] evm: security.ima [ 3.348445] evm: security.capability [ 3.349347] evm: HMAC attrs: 0x1 [ 3.351853] rtc_cmos 00:05: setting system clock to 2026-09-15 13:40:29 UTC (1789479629) [ 3.356714] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 3.359193] debug: unmapping init [mem 0xffffffff8cb82000-0xffffffff8ce58fff] [ 3.368108] Write protecting the kernel read-only data: 28672k [ 3.371170] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 3.373925] debug: unmapping init [mem 0xffffffff8bb14000-0xffffffff8bbfffff] [ 3.406332] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.417323] systemd[1]: Detected virtualization kvm. [ 3.419481] systemd[1]: Detected architecture x86-64. [ 3.421832] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.454755] systemd[1]: No hostname configured. [ 3.457089] systemd[1]: Set hostname to . [ 3.458833] random: systemd: uninitialized urandom read (16 bytes read) [ 3.461151] systemd[1]: Initializing machine ID from random generator. [ 3.502354] random: ln: uninitialized urandom read (6 bytes read) [ 3.579249] random: systemd: uninitialized urandom read (16 bytes read) [ 3.580917] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.584424] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.587269] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.070378] device-mapper: uevent: version 1.0.3 [ 4.071784] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 4.649344] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.731731] virtio_net virtio0 ens2: renamed from eth0 [ 4.751588] scsi host0: ata_piix [ 4.760956] scsi host1: ata_piix [ 4.762570] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.765027] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.321629] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.853793] random: crng init done [ 9.854655] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.060791] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.122156] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.368859] SELinux: Disabled at runtime. [ 11.429299] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.436160] systemd[1]: Detected virtualization kvm. [ 11.438055] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.877168] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.880533] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.885510] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.890282] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.892592] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.898155] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.906243] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK [[ 12.004903] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.269842] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.588882] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.596502] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.751227] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.760991] EDAC sbridge: Ver: 1.1.2 [ 14.106595] Key type dns_resolver registered [ 14.407808] NFS: Registering the id_resolver key type [ 14.409781] Key type id_resolver registered [ 14.411332] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-server login: [ 41.380744] libcfs: loading out-of-tree module taints kernel. [ 41.416712] Key type ._llcrypt registered [ 41.418944] Key type .llcrypt registered [ 41.501930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 60.612484] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 62.679492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.691861] alg: No test for adler32 (adler32-zlib) [ 63.239679] hrtimer: interrupt took 25713039 ns [ 64.262544] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 65.540026] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 67.423683] Key type lgssc registered [ 70.518110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.623311] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.535754] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 145.126204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.176665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.451761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.496981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.690925] Lustre: lustre-MDT0000: new disk, initializing [ 146.880836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.923598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.994721] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.990673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.076947] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.101522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.105502] Lustre: Skipped 1 previous similar message [ 166.185259] Lustre: lustre-MDT0001: new disk, initializing [ 166.254633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.291869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.298772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.753186] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.998207] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.248417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.496941] Lustre: lustre-OST0000: new disk, initializing [ 186.500849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.505867] Lustre: 8466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.569181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.664468] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.606269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.615600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.710276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.639650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.826808] Lustre: lustre-OST0001: new disk, initializing [ 206.835980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.848138] Lustre: 9539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.949832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.695113] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.002785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.011328] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.052646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.771551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.528548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.401980] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.342033] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing yml_node [ 250.297731] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 253.846805] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 256.886188] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 258.974841] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 09:44:44 EDT 2026 [ 276.334183] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 278.093939] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 294.367952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 294.370283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.376663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.891234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.904265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.905301] Lustre: Skipped 2 previous similar messages [ 294.920764] Lustre: Skipped 2 previous similar messages [ 299.999914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.005264] Lustre: Skipped 3 previous similar messages [ 300.854912] Lustre: server umount lustre-MDT0000 complete [ 305.634427] LustreError: 8465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479931 with bad export cookie 15119436858058276994 [ 305.646983] LustreError: 8465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 305.647314] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.239902] LustreError: 8472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.240355] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.264164] LustreError: 8472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 310.305830] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.928875] Lustre: server umount lustre-MDT0001 complete [ 318.722594] Lustre: server umount lustre-OST0000 complete [ 323.946930] Lustre: server umount lustre-OST0001 complete [ 333.876478] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 344.132576] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 385.344577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.724898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 385.763339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.817718] Lustre: lustre-MDT0000: new disk, initializing [ 385.875891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.912493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 390.461553] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 402.398095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.598584] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.658886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.663539] Lustre: Skipped 1 previous similar message [ 402.737290] Lustre: lustre-MDT0001: new disk, initializing [ 402.844252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.906096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 402.916155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.526078] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 419.267276] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.900087] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.871975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.262973] Lustre: lustre-OST0000: new disk, initializing [ 433.271528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.287050] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.430000] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.246339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.272202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.404995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.266994] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 454.045537] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.880509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.112622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.477989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.834251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 471.523970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.524062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.532340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.532350] Lustre: Skipped 1 previous similar message [ 471.552641] Lustre: Skipped 2 previous similar messages [ 481.762730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.773741] Lustre: Skipped 4 previous similar messages [ 483.295148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.595601] Lustre: server umount lustre-OST0000 complete [ 489.959560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.968365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.239137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 502.241967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.263597] Lustre: Skipped 6 previous similar messages [ 502.554513] Lustre: server umount lustre-MDT0000 complete [ 506.464127] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480132 with bad export cookie 15119436858058278527 [ 506.469916] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.475187] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 507.109933] Lustre: server umount lustre-MDT0001 complete [ 511.656713] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:48:57 (1789480137) [ 514.689499] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 516.468665] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:49:02 (1789480142) [ 524.919278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.531417] LustreError: 19473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.614677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.791725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 530.919497] LustreError: 19474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.040553] LustreError: 19473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.133606] LustreError: 19474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.891490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.171384] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 556.139229] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.944929] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.319750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.781854] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 589.434228] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.689763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.697389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.731503] Lustre: Skipped 1 previous similar message [ 606.742427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.289903] Lustre: server umount lustre-OST0000 complete [ 613.362224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 627.167379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 627.706640] Lustre: server umount lustre-MDT0000 complete [ 630.240753] LustreError: 20201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.274634] LustreError: 20201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 632.769952] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480258 with bad export cookie 15119436858058279402 [ 632.785198] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 632.787870] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.277949] Lustre: server umount lustre-MDT0001 complete [ 647.565693] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 650.824070] Key type lgssc unregistered [ 651.254079] LNet: 23027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.258716] LNetError: 23027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.338523] LNet: Removed LNI 192.168.201.113@tcp [ 653.526128] Key type .llcrypt unregistered [ 653.529661] Key type ._llcrypt unregistered [ 676.649435] Key type ._llcrypt registered [ 676.651544] Key type .llcrypt registered [ 676.749615] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 677.774594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.875920] alg: No test for adler32 (adler32-zlib) [ 678.892389] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 679.078064] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 680.713494] Key type lgssc registered [ 682.258570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.475853] loop: module loaded [ 692.553234] loop0: detected capacity change from 0 to 204800000 [ 692.757815] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.777306] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.344489] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.088433] loop0: detected capacity change from 0 to 204800000 [ 701.346974] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.379886] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.404803] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.833380] loop0: detected capacity change from 0 to 204800000 [ 713.964842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.002627] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.442339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 715.535134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 715.667618] Lustre: lustre-MDT0000: new disk, initializing [ 715.792185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.819077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.480989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 732.006405] loop1: detected capacity change from 0 to 204800000 [ 740.000860] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 740.431323] Lustre: lustre-OST07c6: new disk, initializing [ 740.437376] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 740.448620] Lustre: Skipped 1 previous similar message [ 740.453656] Lustre: 26217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 740.591854] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 748.146807] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 750.279995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 750.298232] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 750.372440] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 754.159791] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 754.177845] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 754.197648] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 754.211370] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 754.224597] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 754.232621] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 772.799919] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 772.808989] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.823581] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.837577] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 772.846695] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.858813] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.697326] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 778.709884] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 778.725490] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 778.736476] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.751947] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 778.764514] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.772498] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 778.782213] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.790410] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 778.797839] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.806497] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.816238] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.144327] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 782.164593] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 782.175660] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 782.188463] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.194951] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.202848] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.210715] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 782.220962] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.239721] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 782.253511] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.273012] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.286686] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.569512] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 787.584131] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 787.593186] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.602598] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.616147] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.623218] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.635736] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 787.649535] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.661698] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 787.673232] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.689140] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 787.699452] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.049367] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 796.062302] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 796.067099] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 796.076296] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.088229] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 796.098908] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.113867] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 796.123726] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.134042] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 796.141704] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.149868] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 796.156235] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 812.784189] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 812.790946] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 812.795492] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 812.802426] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 812.809789] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.814616] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 812.819702] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 812.827958] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 812.834200] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 812.840126] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 812.845052] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.852973] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 852.449989] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.471608] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 857.578411] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 862.694667] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 863.199577] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 863.424781] Lustre: server umount lustre-OST07c6 complete [ 874.642887] Lustre: server umount lustre-MDT0000 complete [ 893.719744] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 896.434546] Key type lgssc unregistered [ 896.792958] LNet: 28777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.804606] LNetError: 28777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.819950] LNet: Removed LNI 192.168.201.113@tcp [ 898.032481] Key type .llcrypt unregistered [ 898.039609] Key type ._llcrypt unregistered [ 923.689520] Key type ._llcrypt registered [ 923.691723] Key type .llcrypt registered [ 923.835309] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 925.728700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.758828] alg: No test for adler32 (adler32-zlib) [ 927.031304] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 927.476309] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 929.239417] Key type lgssc registered [ 930.669223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.534982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.564745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.246646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.388179] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 951.778392] LustreError: 30047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.786959] LustreError: 30047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 956.913171] LustreError: 30046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.543491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.006210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.282223] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 973.095713] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 979.849455] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 986.422575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.920453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.581118] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1001.344610] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.516254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1029.090499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.098029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.119948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.555506] Lustre: server umount lustre-OST0000 complete [ 1034.721835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1034.735512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.758146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.282278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.283674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.401638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.410236] Lustre: Skipped 2 previous similar messages [ 1043.588709] Lustre: server umount lustre-MDT0000 complete [ 1047.196604] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480673 with bad export cookie 18036086355995875349 [ 1047.205273] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.218121] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1047.699376] Lustre: server umount lustre-MDT0001 complete [ 1058.831498] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1062.052509] Key type lgssc unregistered [ 1062.335948] LNet: 33806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.349320] LNetError: 33806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.397632] LNet: Removed LNI 192.168.201.113@tcp [ 1064.234426] Key type .llcrypt unregistered [ 1064.236511] Key type ._llcrypt unregistered [ 1080.345625] Key type ._llcrypt registered [ 1080.351061] Key type .llcrypt registered [ 1080.545890] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1081.631332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.669027] alg: No test for adler32 (adler32-zlib) [ 1082.794966] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1083.041863] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1084.687200] Key type lgssc registered [ 1085.823912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.158742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1094.177241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.506073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.441294] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1104.878681] LustreError: 35052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.896407] LustreError: 35052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1109.106988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.509372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1114.265791] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1121.554782] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1127.495613] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1134.611610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.091930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.231066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1141.008477] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1149.159901] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1164.770302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.771710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.788061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.800364] Lustre: Skipped 1 previous similar message [ 1169.324967] Lustre: server umount lustre-OST0000 complete [ 1175.009457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.013868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.028151] Lustre: Skipped 2 previous similar messages [ 1179.314216] Lustre: server umount lustre-MDT0000 complete [ 1180.133268] LustreError: 35053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.167492] LustreError: 35053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1183.454189] LustreError: 35033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480809 with bad export cookie 16653570145971339435 [ 1183.471942] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.805493] Lustre: server umount lustre-MDT0001 complete [ 1195.490336] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1198.095549] Key type lgssc unregistered [ 1198.392837] LNet: 38605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.406837] LNetError: 38605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.471543] LNet: Removed LNI 192.168.201.113@tcp [ 1200.278466] Key type .llcrypt unregistered [ 1200.280245] Key type ._llcrypt unregistered [ 1223.093985] Key type ._llcrypt registered [ 1223.096507] Key type .llcrypt registered [ 1223.178383] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1224.050528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.222421] alg: No test for adler32 (adler32-zlib) [ 1225.269509] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1225.469498] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1227.119986] Key type lgssc registered [ 1228.322858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.838518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.865239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.231914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.829769] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1247.712564] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.741677] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1250.197936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.655063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.062073] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1262.533677] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1269.036720] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1276.178374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.522385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.494760] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1282.674899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1288.410313] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1300.449974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.454825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.469843] Lustre: Skipped 1 previous similar message [ 1300.479800] Lustre: Skipped 2 previous similar messages [ 1301.462508] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1304.880865] Lustre: server umount lustre-MDT0000 complete [ 1305.575026] LustreError: 42524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.579938] LustreError: 42524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1308.246610] LustreError: 42610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480934 with bad export cookie 4253975650718889474 [ 1308.250487] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.254485] LustreError: 42610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.698267] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.699845] LustreError: 40026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.729410] Lustre: Skipped 1 previous similar message [ 1310.753476] LustreError: 40026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1310.773559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1313.696964] LustreError: 40027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.697076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1313.726447] LustreError: 40027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1313.745132] Lustre: Skipped 1 previous similar message [ 1314.705713] Lustre: server umount lustre-MDT0001 complete [ 1335.813477] Lustre: server umount lustre-OST0000 complete [ 1351.936986] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1354.800590] Key type lgssc unregistered [ 1355.332663] LNet: 43891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.342461] LNetError: 43891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.362224] LNet: Removed LNI 192.168.201.113@tcp [ 1356.476281] Key type .llcrypt unregistered [ 1356.477653] Key type ._llcrypt unregistered [ 1378.120955] Key type ._llcrypt registered [ 1378.123819] Key type .llcrypt registered [ 1378.229960] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1379.812283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.836411] alg: No test for adler32 (adler32-zlib) [ 1380.865867] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1381.141840] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1382.785481] Key type lgssc registered [ 1383.977646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.978534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.002940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.383664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.302588] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1403.874826] LustreError: 45304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.879524] LustreError: 45304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1406.258124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.603641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.214103] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1418.214594] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1423.440710] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.135891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.396663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.153383] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1439.610843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1442.946949] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1449.954592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.969104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.980352] Lustre: Skipped 1 previous similar message [ 1450.004429] Lustre: Skipped 1 previous similar message [ 1450.985458] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1455.079918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.089924] Lustre: Skipped 1 previous similar message [ 1455.824540] Lustre: server umount lustre-OST0000 complete [ 1475.557122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.567979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.576450] Lustre: Skipped 1 previous similar message [ 1475.590507] Lustre: Skipped 1 previous similar message [ 1479.428909] Lustre: server umount lustre-MDT0000 complete [ 1480.673579] LustreError: 45304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.691445] LustreError: 45304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1481.696902] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.724108] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1482.527572] LustreError: 48259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481108 with bad export cookie 12200773774680000485 [ 1482.528716] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.538025] LustreError: 48259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.127281] Lustre: server umount lustre-MDT0001 complete [ 1498.169354] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1500.735181] Key type lgssc unregistered [ 1501.086600] LNet: 49010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.099082] LNetError: 49010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.124340] LNet: Removed LNI 192.168.201.113@tcp [ 1502.009162] Key type .llcrypt unregistered [ 1502.011133] Key type ._llcrypt unregistered [ 1524.902314] Key type ._llcrypt registered [ 1524.904062] Key type .llcrypt registered [ 1525.066724] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1526.130965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.241097] alg: No test for adler32 (adler32-zlib) [ 1527.252469] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1527.486631] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1529.143227] Key type lgssc registered [ 1530.287753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1538.157000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1538.184071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.702093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.003886] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1549.295737] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1549.313415] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1551.787985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.111020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.087201] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1563.598213] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1568.836750] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1575.290401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.750257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.822059] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1583.926835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1589.150768] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1594.239610] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1597.354604] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1609.636126] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1510h07m14s [ 1611.259951] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1613.799513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.810800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.826129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.885372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.895142] Lustre: Skipped 4 previous similar messages [ 1616.368203] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1619.598916] Lustre: server umount lustre-MDT0000 complete [ 1620.963213] LustreError: 50413:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.987866] LustreError: 50413:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1622.479590] LustreError: 52205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.173723] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1627.105788] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.120644] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1628.869398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.981817] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.161267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.203434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1630.188799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1633.491576] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1636.669892] Lustre: DEBUG MARKER: Wait for df (36693) ... [ 1638.317983] Lustre: DEBUG MARKER: done [ 1644.512898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.515185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.523777] LustreError: Skipped 1 previous similar message [ 1644.525494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1644.525501] Lustre: Skipped 1 previous similar message [ 1644.545391] Lustre: Skipped 3 previous similar messages [ 1650.658700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.672811] Lustre: Skipped 4 previous similar messages [ 1655.263674] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1655.414049] Lustre: server umount lustre-OST0000 complete [ 1660.897371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.906142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.352416] Lustre: server umount lustre-MDT0000 complete [ 1668.065382] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.080575] LustreError: 50417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1668.743482] LustreError: 50398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481294 with bad export cookie 2592176219287222916 [ 1668.746345] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.752230] LustreError: 50398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1668.955565] Lustre: server umount lustre-MDT0001 complete [ 1678.356156] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1680.598816] Key type lgssc unregistered [ 1680.845218] LNet: 55340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.852320] LNetError: 55340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.894313] LNet: Removed LNI 192.168.201.113@tcp [ 1682.629289] Key type .llcrypt unregistered [ 1682.631200] Key type ._llcrypt unregistered [ 1718.450620] Key type ._llcrypt registered [ 1718.453280] Key type .llcrypt registered [ 1718.549044] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1719.632863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.668762] alg: No test for adler32 (adler32-zlib) [ 1720.729825] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1720.934393] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1722.607287] Key type lgssc registered [ 1723.639835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.304459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.312851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.538805] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1731.566545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1731.677141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.533186] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1742.845125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.916368] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1742.932836] Lustre: 57546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.073273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.134658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1752.283825] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1756.851359] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.035879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.141737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1762.284669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.792490] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1771.080656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1772.561664] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.532580] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1510h10m05s [ 1781.039668] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1788.040474] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1790.190167] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1790.472239] LustreError: 57550:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1810.915292] Lustre: lustre-MDT0000: Client 24aa3318-50df-4808-bf8f-19572d94fcb8 (at 192.168.201.13@tcp) reconnecting [ 1830.576291] LustreError: 57550:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1832.379212] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1834.700984] Lustre: DEBUG MARKER: done [ 1842.659386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.663962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1842.685660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.696543] Lustre: Skipped 1 previous similar message [ 1845.130994] Lustre: server umount lustre-OST0000 complete [ 1850.849276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1850.861083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.870230] Lustre: Skipped 1 previous similar message [ 1850.881500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.937157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.940376] Lustre: Skipped 1 previous similar message [ 1853.985986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.991262] Lustre: Skipped 1 previous similar message [ 1854.780821] Lustre: server umount lustre-MDT0000 complete [ 1857.361220] LustreError: 57536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481483 with bad export cookie 13985024389345196041 [ 1857.365906] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.626214] Lustre: server umount lustre-MDT0001 complete [ 1865.932070] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1867.896832] Key type lgssc unregistered [ 1868.154411] LNet: 61875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.163526] LNetError: 61875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.227496] LNet: Removed LNI 192.168.201.113@tcp [ 1870.001552] Key type .llcrypt unregistered [ 1870.004083] Key type ._llcrypt unregistered [ 1907.403594] Key type ._llcrypt registered [ 1907.407670] Key type .llcrypt registered [ 1907.529554] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1908.435600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.578861] alg: No test for adler32 (adler32-zlib) [ 1909.633796] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1909.805993] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1911.431237] Key type lgssc registered [ 1912.278410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1914.750429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1914.769095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.961189] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.975669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1916.077141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.770726] Lustre: Failing over lustre-MDT0000 [ 1916.938171] Lustre: server umount lustre-MDT0000 complete [ 1922.006804] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:12:27 (1789481547) [ 1927.772543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.120478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.858819] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1938.852474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.947280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1938.996832] Lustre: 64648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1939.203785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.472695] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1948.120565] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.486264] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.755224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.905947] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1958.061750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.634964] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1964.179764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1969.087281] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1974.891507] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1980.331582] Lustre: server umount lustre-MDT0000 complete [ 1980.387177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1980.391813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.523652] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481609 with bad export cookie 10334275505224780001 [ 1983.544751] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1983.553892] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.001995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.032854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1985.867364] Lustre: server umount lustre-MDT0001 complete [ 1989.259364] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1993.058299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2001.334117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.489988] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f6ab8d193f191c4 [ 2008.510598] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2008.768826] LustreError: 66495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.780506] LustreError: 66495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2008.895515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.013670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2009.891373] LustreError: 66496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.071676] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2015.206345] LustreError: 66495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.323284] LustreError: 66496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.581914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.880622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.974899] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.498743] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2031.207387] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.850986] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.313108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2041.319276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.329035] Lustre: Skipped 1 previous similar message [ 2041.339485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.686528] Lustre: server umount lustre-MDT0000 complete [ 2044.387390] LustreError: 66496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.399691] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.005789] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481673 with bad export cookie 10334275505224782276 [ 2047.009535] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.013426] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.319648] Lustre: server umount lustre-MDT0001 complete [ 2053.820643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2061.869115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.867978] Lustre: 63570:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481675/real 1789481675] req@00000000bf5227eb x1876407390107264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789481691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.888662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.901541] Lustre: Skipped 1 previous similar message [ 2067.872252] Lustre: 63568:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481678/real 1789481678] req@00000000f51c8b48 x1876407390107392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789481694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.983719] Lustre: 63568:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481680/real 1789481680] req@0000000097692e2a x1876407390107520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789481696 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.373290] LustreError: 66496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.391703] LustreError: 66496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2072.436874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.486156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2073.439357] Lustre: 63571:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481683/real 1789481683] req@00000000c89f13d6 x1876407390107776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789481699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2076.891709] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2078.698748] Lustre: 63570:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789481688/real 1789481688] req@000000008092a873 x1876407390107904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789481704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.854312] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2084.865348] Lustre: Skipped 1 previous similar message [ 2085.152566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.447269] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2090.477228] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2096.144091] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2100.869831] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.302623] Lustre: server umount lustre-MDT0000 complete [ 2110.948412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.952280] LustreError: 71095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.957199] Lustre: Skipped 2 previous similar messages [ 2110.982447] LustreError: 71095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2111.599777] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481737 with bad export cookie 10334275505224783655 [ 2111.606909] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.610083] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2111.860280] Lustre: server umount lustre-MDT0001 complete [ 2117.024120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2120.238280] Lustre: DEBUG MARKER: files compared the same [ 2131.076423] Lustre: server umount lustre-OST0000 complete [ 2153.897860] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2157.741721] Key type lgssc unregistered [ 2158.117066] LNet: 74390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.129974] LNetError: 74390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.174221] LNet: Removed LNI 192.168.201.113@tcp [ 2159.768519] Key type .llcrypt unregistered [ 2159.772694] Key type ._llcrypt unregistered [ 2193.025946] Key type ._llcrypt registered [ 2193.030560] Key type .llcrypt registered [ 2193.264071] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2195.557443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.718401] alg: No test for adler32 (adler32-zlib) [ 2197.502816] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2197.885528] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2199.615220] Key type lgssc registered [ 2201.068708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2209.712395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2209.727298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.175501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.028270] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2221.539643] LustreError: 75807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.550565] LustreError: 75807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2226.666726] LustreError: 75808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.729468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.112984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.429904] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2244.350838] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2254.468765] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2265.033990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.548486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.487031] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2279.927242] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.890671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2293.216053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2293.219705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.229381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2296.289745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.295507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.307177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.254319] Lustre: server umount lustre-OST0000 complete [ 2303.967982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2303.978173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.000690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.006983] Lustre: Skipped 1 previous similar message [ 2306.528258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.531483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.653143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.667200] Lustre: Skipped 2 previous similar messages [ 2318.303165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.654991] Lustre: server umount lustre-MDT0000 complete [ 2320.870159] LustreError: 75808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.890861] LustreError: 75808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2322.850846] LustreError: 75790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481948 with bad export cookie 12872487337233401871 [ 2322.862051] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.409873] Lustre: server umount lustre-MDT0001 complete [ 2335.989626] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2339.120506] Key type lgssc unregistered [ 2339.459229] LNet: 79364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.469364] LNetError: 79364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.484673] LNet: Removed LNI 192.168.201.113@tcp [ 2340.391163] Key type .llcrypt unregistered [ 2340.395228] Key type ._llcrypt unregistered [ 2370.942859] Key type ._llcrypt registered [ 2370.944903] Key type .llcrypt registered [ 2371.095680] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2372.474112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2372.691332] alg: No test for adler32 (adler32-zlib) [ 2373.876587] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2374.092585] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2375.733934] Key type lgssc registered [ 2377.653810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.473159] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.486526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2400.991561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2406.114116] LustreError: 80752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 2432.863811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.570472] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2447.745780] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2456.176431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.316384] Lustre: *** cfs_fail_loc=706, val=0*** [ 2456.688686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.773901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2461.186429] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2462.198256] LustreError: 80777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.229025] LustreError: 80777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2467.307448] LustreError: 80778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.345197] LustreError: 81838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.365931] LustreError: 81838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2471.251476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.607560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.838329] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2485.742420] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2492.484398] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2497.512801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.544765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2497.545389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.568468] Lustre: Skipped 1 previous similar message [ 2502.629553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.749140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.760016] Lustre: Skipped 2 previous similar messages [ 2509.279202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2509.423947] Lustre: server umount lustre-OST0000 complete [ 2517.985724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.987936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.998079] LustreError: Skipped 1 previous similar message [ 2518.008655] Lustre: Skipped 1 previous similar message [ 2518.009817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.286204] Lustre: server umount lustre-MDT0000 complete [ 2523.120377] LustreError: 81842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.140775] LustreError: 81842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2524.567390] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482150 with bad export cookie 2269269528020585811 [ 2524.580288] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.588605] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2524.990897] Lustre: server umount lustre-MDT0001 complete [ 2538.345610] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2542.354399] Key type lgssc unregistered [ 2542.782780] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.795810] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.818398] LNet: Removed LNI 192.168.201.113@tcp [ 2544.003127] Key type .llcrypt unregistered [ 2544.004803] Key type ._llcrypt unregistered [ 2573.499458] Key type ._llcrypt registered [ 2573.502038] Key type .llcrypt registered [ 2573.603166] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2575.022981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2575.059521] alg: No test for adler32 (adler32-zlib) [ 2576.191640] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2576.535369] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2578.239279] Key type lgssc registered [ 2579.499305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.331825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.356751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.884641] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2603.556477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.249111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2609.637755] LustreError: 86478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.662514] LustreError: 86478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2609.915303] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2614.759682] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.774510] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2619.873058] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.900125] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2620.024659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.408498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.512516] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.526528] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2626.900865] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2629.603263] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.640204] LustreError: 86482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2637.323905] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2637.334314] Lustre: MGS: Client 2bc34178-1f81-456a-90a1-4e231b3220a9 (at 0@lo) reconnecting [ 2637.356714] LustreError: 86492:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2637.627293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.709288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2642.713714] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2658.279856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.285684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.297983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.316735] Lustre: Skipped 1 previous similar message [ 2662.252307] Lustre: server umount lustre-OST0000 complete [ 2668.512431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.517175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.546666] Lustre: Skipped 1 previous similar message [ 2668.547273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.569241] Lustre: Skipped 2 previous similar messages [ 2673.632161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.661458] Lustre: Skipped 1 previous similar message [ 2678.766540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.311225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2681.426041] Lustre: server umount lustre-MDT0000 complete [ 2683.877435] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.891373] LustreError: 86481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2685.642244] Lustre: server umount lustre-MDT0001 complete [ 2689.885058] Lustre: server umount MGS complete [ 2705.080508] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:25:29 (1789482329) [ 2738.532347] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 2748.146656] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2789.584524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.339097] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2804.666887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.019663] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2805.090986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2805.229662] Lustre: lustre-MDT0001: new disk, initializing [ 2805.351965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2805.403641] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2805.425844] Lustre: 93651:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2805.445989] LustreError: 93651:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2810.234904] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2822.325700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.824812] Lustre: lustre-OST0000: new disk, initializing [ 2822.851154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2822.869826] Lustre: 94417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2823.007779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.984222] Lustre: 94651:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006d44d571 x1876408089486592/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789482472 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2831.378634] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2842.607588] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2849.756763] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.773233] LustreError: 85147:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000015dbb687 x1876408089488128/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2849.821332] Lustre: MGS: Client 6361bfcb-8b58-4c5c-a51d-3fd2861c8a9e (at 0@lo) reconnecting [ 2849.826413] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbabb426bf34b9c55 [ 2849.835706] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2849.849990] LustreError: 93637:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2849.947219] Lustre: 92919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2850.020309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2850.145404] Lustre: lustre-MDT0000: new disk, initializing [ 2850.274117] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2850.300022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.374039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2851.488134] Lustre: 95488:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac525b87 x1876408089496448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789482493 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2855.991384] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2856.675953] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2860.596438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2860.620120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2860.758967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2884.069659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.071955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.077722] Lustre: Skipped 1 previous similar message [ 2884.101624] Lustre: Skipped 2 previous similar messages [ 2889.775617] Lustre: server umount lustre-OST0000 complete [ 2894.305230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.310973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.334291] Lustre: Skipped 1 previous similar message [ 2894.336336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.342254] Lustre: Skipped 3 previous similar messages [ 2908.127330] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.393278] Lustre: server umount lustre-MDT0000 complete [ 2909.666169] LustreError: 96026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.688489] LustreError: 96026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2920.333290] Lustre: server umount lustre-MDT0001 complete [ 2925.444734] Lustre: server umount MGS complete [ 2938.038341] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:29:23 (1789482563) [ 2954.876699] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2957.596644] Key type lgssc unregistered [ 2957.814292] LNet: 97987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2957.821163] LNetError: 97987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2957.842594] LNet: Removed LNI 192.168.201.113@tcp [ 2958.721945] Key type .llcrypt unregistered [ 2958.724993] Key type ._llcrypt unregistered [ 2978.117577] Key type ._llcrypt registered [ 2978.121558] Key type .llcrypt registered [ 2978.300236] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2979.426322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2979.600431] alg: No test for adler32 (adler32-zlib) [ 2980.744355] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2980.989328] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2982.639227] Key type lgssc registered [ 2983.848778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2991.646653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2991.663281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.237692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.328729] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3003.372114] LustreError: 99205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.398169] LustreError: 99205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3006.584630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.972375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.214523] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3020.806866] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3026.669854] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3033.213996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.574238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.722514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3040.962844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3048.814667] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3061.217221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3061.225760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.233841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.254603] Lustre: Skipped 1 previous similar message [ 3066.089957] Lustre: server umount lustre-OST0000 complete [ 3071.458348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.471064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3071.478451] Lustre: Skipped 1 previous similar message [ 3071.482313] Lustre: Skipped 1 previous similar message [ 3076.580238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.590464] Lustre: Skipped 2 previous similar messages [ 3081.702995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.711675] Lustre: Skipped 1 previous similar message [ 3084.256223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3084.526087] Lustre: server umount lustre-MDT0000 complete [ 3086.817346] LustreError: 101511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.848635] LustreError: 101511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3088.194217] LustreError: 99185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482714 with bad export cookie 16508048106661296033 [ 3088.199153] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.201124] LustreError: 99185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3088.600192] Lustre: server umount lustre-MDT0001 complete [ 3102.330217] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3106.571801] Key type lgssc unregistered [ 3107.086804] LNet: 102739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3107.097754] LNetError: 102739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3108.133821] LNet: Removed LNI 192.168.201.113@tcp [ 3109.264953] Key type .llcrypt unregistered [ 3109.266994] Key type ._llcrypt unregistered [ 3125.715957] Key type ._llcrypt registered [ 3125.719207] Key type .llcrypt registered [ 3125.835755] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3126.895968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3127.010906] alg: No test for adler32 (adler32-zlib) [ 3128.079645] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3128.472701] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3130.198061] Key type lgssc registered [ 3131.983683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3138.951831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3138.958950] LustreError: 103818:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3138.982315] LustreError: 103819:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3138.986360] LustreError: 103818:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3138.989081] LustreError: 103818:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3139.001385] LustreError: 103818:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3139.010789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.585096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.479503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.831340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3154.007504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3160.294411] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lsmod [ 3163.025440] LustreError: 105018:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3163.096848] LustreError: 105025:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3163.107971] LustreError: 105018:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4923 [ 3163.116600] LustreError: 105018:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3163.130853] LustreError: 105018:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3163.135719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.655249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.693738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3169.248766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3169.263604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.270952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.789200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.794174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.803797] Lustre: Skipped 1 previous similar message [ 3170.844298] Lustre: Skipped 1 previous similar message [ 3172.749288] Lustre: server umount lustre-MDT0000 complete [ 3172.907195] LustreError: 103852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.916120] LustreError: 103852:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3177.119806] LustreError: 103831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482803 with bad export cookie 9195714529787153925 [ 3177.143213] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.149700] LustreError: 103831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3177.957699] LustreError: 103853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.966382] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.992642] LustreError: 103853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3178.043401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3178.054687] Lustre: Skipped 1 previous similar message [ 3179.103303] LustreError: 104287:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000053ef628e x1876408668280320/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3179.408079] Lustre: server umount lustre-MDT0001 complete [ 3183.632376] Lustre: server umount lustre-OST0000 complete [ 3193.391531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.838041] LustreError: 106087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.936533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.937229] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3198.944488] LustreError: 106088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.065696] LustreError: 106087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.242717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.583975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.027357] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3222.681311] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3230.385065] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3238.333531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.873975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.931902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3245.912341] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3254.860918] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3264.490960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.492105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3264.492301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.513361] Lustre: Skipped 1 previous similar message [ 3267.761816] Lustre: server umount lustre-OST0000 complete [ 3274.741478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.747123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.756311] Lustre: Skipped 1 previous similar message [ 3274.769728] Lustre: Skipped 2 previous similar messages [ 3278.107164] Lustre: server umount lustre-MDT0000 complete [ 3279.841071] LustreError: 107521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.868144] LustreError: 107521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3281.376521] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482907 with bad export cookie 9195714529787154849 [ 3281.387681] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3281.395749] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.650428] Lustre: server umount lustre-MDT0001 complete [ 3291.394125] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3294.293965] Key type lgssc unregistered [ 3294.576473] LNet: 109624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3294.583631] LNetError: 109624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3294.604265] LNet: Removed LNI 192.168.201.113@tcp [ 3295.361136] Key type .llcrypt unregistered [ 3295.363723] Key type ._llcrypt unregistered [ 3318.194812] Key type ._llcrypt registered [ 3318.196875] Key type .llcrypt registered [ 3318.297273] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3319.495871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3319.596325] alg: No test for adler32 (adler32-zlib) [ 3320.610613] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3320.846764] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3322.535313] Key type lgssc registered [ 3323.763921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3331.562197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3331.596326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.090392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.128737] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3343.334152] LustreError: 111038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.360492] LustreError: 111038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3344.935225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.223392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.941851] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3358.018288] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3363.754700] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3371.320405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.809682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.878766] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3377.968052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3385.871436] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.626390] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3399.756584] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3405.801018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.802812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.821929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.830437] Lustre: Skipped 1 previous similar message [ 3410.823336] Lustre: server umount lustre-OST0000 complete [ 3416.035854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.037412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.061828] Lustre: Skipped 1 previous similar message [ 3416.084743] Lustre: Skipped 2 previous similar messages [ 3421.152447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.167241] Lustre: Skipped 1 previous similar message [ 3426.278490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.283902] Lustre: Skipped 1 previous similar message [ 3429.343731] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3429.830495] Lustre: server umount lustre-MDT0000 complete [ 3431.394374] LustreError: 111037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.409102] LustreError: 111037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3434.003888] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483060 with bad export cookie 12403149770316033889 [ 3434.005904] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.018195] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3434.366326] Lustre: server umount lustre-MDT0001 complete [ 3445.536425] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3448.088632] Key type lgssc unregistered [ 3448.388911] LNet: 114675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3448.399231] LNetError: 114675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3448.424406] LNet: Removed LNI 192.168.201.113@tcp [ 3449.512675] Key type .llcrypt unregistered [ 3449.517829] Key type ._llcrypt unregistered [ 3468.634547] Key type ._llcrypt registered [ 3468.641143] Key type .llcrypt registered [ 3468.793277] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3470.185331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3470.307216] alg: No test for adler32 (adler32-zlib) [ 3471.758851] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3471.935898] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3473.647230] Key type lgssc registered [ 3475.279371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3483.980669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3483.997705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.527771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.000120] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3494.885499] LustreError: 115893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.898089] LustreError: 115893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3498.427817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.845493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3503.828640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3511.962799] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3518.441275] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3526.422043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.848446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.935113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3533.496191] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3542.173895] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3553.761829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.767669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.771513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.775028] Lustre: Skipped 1 previous similar message [ 3553.782138] Lustre: Skipped 1 previous similar message [ 3557.699837] Lustre: server umount lustre-OST0000 complete [ 3564.007129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.020619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.033154] Lustre: Skipped 1 previous similar message [ 3569.122542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3569.130045] Lustre: Skipped 1 previous similar message [ 3574.242655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3574.256831] Lustre: Skipped 1 previous similar message [ 3576.287133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.525748] Lustre: server umount lustre-MDT0000 complete [ 3579.369047] LustreError: 115893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.388381] LustreError: 115893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3580.279045] LustreError: 118475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483206 with bad export cookie 18400243521898726141 [ 3580.281215] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.295270] LustreError: 118475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3580.653993] Lustre: server umount lustre-MDT0001 complete [ 3593.047549] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3596.332533] Key type lgssc unregistered [ 3596.610554] LNet: 119430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.617279] LNetError: 119430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3596.627630] LNet: Removed LNI 192.168.201.113@tcp [ 3597.483121] Key type .llcrypt unregistered [ 3597.484963] Key type ._llcrypt unregistered [ 3621.339524] Key type ._llcrypt registered [ 3621.342813] Key type .llcrypt registered [ 3621.444967] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3622.713697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3622.957962] alg: No test for adler32 (adler32-zlib) [ 3623.960555] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3624.177473] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3625.880249] Key type lgssc registered [ 3627.186854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.805590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.824021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.318754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.432647] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3647.459226] LustreError: 120842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.478798] LustreError: 120842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3652.152082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.692588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3658.779082] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3667.621830] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.902928] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3681.396634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.811243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.818242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3688.596948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3695.743692] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.931947] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3706.384392] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3706.389166] Lustre: Skipped 2 previous similar messages [ 3708.984479] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3709.030670] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3710.570617] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3710.585353] Lustre: Skipped 2 previous similar messages [ 3714.290045] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3714.293652] Lustre: Skipped 1 previous similar message [ 3719.689564] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3719.699545] Lustre: Skipped 2 previous similar messages [ 3720.313505] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp [ 3731.818312] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.13/24@tcp in log lustre-MDT0000 [ 3731.824988] Lustre: Skipped 2 previous similar messages [ 3740.759931] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 3740.767789] Lustre: Skipped 1 previous similar message [ 3748.322328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.323673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.344551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.402094] Lustre: Skipped 1 previous similar message [ 3750.193978] Lustre: server umount lustre-OST0000 complete [ 3754.978319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3754.984852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.999881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3755.013947] Lustre: Skipped 1 previous similar message [ 3758.593743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.607085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.769931] Lustre: server umount lustre-MDT0000 complete [ 3763.688317] LustreError: 120843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.712586] LustreError: 120843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3765.139798] LustreError: 120822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483391 with bad export cookie 3047148287887003112 [ 3765.143038] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.148855] LustreError: 120822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3765.754609] Lustre: server umount lustre-MDT0001 complete [ 3777.197892] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3781.086814] Key type lgssc unregistered [ 3781.341506] LNet: 125740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3781.352376] LNetError: 125740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3781.372194] LNet: Removed LNI 192.168.201.113@tcp [ 3782.177298] Key type .llcrypt unregistered [ 3782.179262] Key type ._llcrypt unregistered [ 3812.514151] Key type ._llcrypt registered [ 3812.519435] Key type .llcrypt registered [ 3812.680599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3813.179747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.315455] alg: No test for adler32 (adler32-zlib) [ 3814.393513] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3814.582454] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3816.207289] Key type lgssc registered [ 3817.931927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.489550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.507730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.100209] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3830.154893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.156943] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.13/24@tcp [ 3836.253957] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3840.480818] LustreError: 127310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3840.487805] LustreError: 127310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3845.607209] LustreError: 127309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.849156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.398444] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3847.415456] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 3847.462531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3852.958303] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3861.676920] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3868.481723] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3876.084945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3876.579442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.046478] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3892.919388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.166585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3915.748519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3915.751991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.755383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.768779] Lustre: Skipped 1 previous similar message [ 3915.785530] Lustre: Skipped 1 previous similar message [ 3920.551456] Lustre: server umount lustre-OST0000 complete [ 3924.448641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.455530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.468717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.988714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.989752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.383767] Lustre: server umount lustre-MDT0000 complete [ 3931.104604] LustreError: 127310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.119765] LustreError: 127310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3934.179038] LustreError: 127291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483560 with bad export cookie 13199456709407262966 [ 3934.189683] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.200719] LustreError: 127291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3934.439574] Lustre: server umount lustre-MDT0001 complete [ 3946.018258] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3949.146527] Key type lgssc unregistered [ 3949.464849] LNet: 130921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3949.469897] LNetError: 130921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3950.510816] LNet: Removed LNI 192.168.201.113@tcp [ 3951.562363] Key type .llcrypt unregistered [ 3951.566247] Key type ._llcrypt unregistered [ 3983.154672] Key type ._llcrypt registered [ 3983.158866] Key type .llcrypt registered [ 3983.284689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4001.127859] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4002.368858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4002.409726] alg: No test for adler32 (adler32-zlib) [ 4003.829684] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4004.141089] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4005.879224] Key type lgssc registered [ 4007.237712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.906914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.939923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.226812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4050.266959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4050.370866] Lustre: lustre-MDT0000: new disk, initializing [ 4050.462085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.484486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4055.867646] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4067.555231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.687505] Lustre: 134318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4067.772251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4067.790913] Lustre: Skipped 1 previous similar message [ 4067.937494] Lustre: lustre-MDT0001: new disk, initializing [ 4068.063321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4068.116574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4068.132133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4073.090658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4083.772261] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4089.155643] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4096.308118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.616678] Lustre: lustre-OST0000: new disk, initializing [ 4096.623473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4096.627188] Lustre: 136289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4096.719299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.788577] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4104.299515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4104.307634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4104.499267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4114.158331] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4120.329837] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4120.501322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4125.597076] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4125.835596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4130.274996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.275659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.295223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.316618] LustreError: Skipped 1 previous similar message [ 4130.328819] Lustre: Skipped 1 previous similar message [ 4135.394679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.402128] Lustre: Skipped 1 previous similar message [ 4140.521126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4140.526155] Lustre: Skipped 1 previous similar message [ 4142.047199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4142.186717] Lustre: server umount lustre-OST0000 complete [ 4150.240886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4150.253349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4150.274488] Lustre: Skipped 1 previous similar message [ 4150.293205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.092707] Lustre: server umount lustre-MDT0000 complete [ 4157.084441] LustreError: 134313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483783 with bad export cookie 6884427583777789074 [ 4157.087561] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.098639] LustreError: 134313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4157.360679] Lustre: server umount lustre-MDT0001 complete [ 4166.433443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.944278] LustreError: 138507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.962386] LustreError: 138507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4167.008951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.684903] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4172.256588] LustreError: 138508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.381939] LustreError: 138507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.972121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.349613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4185.785252] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4193.662248] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4200.449269] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4209.449520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.818616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.159893] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4224.779155] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4230.887300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4231.176805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.412110] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4237.685271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4247.491026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.758785] Lustre: lustre-OST0001: new disk, initializing [ 4247.764226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4247.779912] Lustre: 141881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.894244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4254.860348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4254.887298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4255.095413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4255.394849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4264.486881] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4264.706844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4268.780658] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4268.955648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.646913] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92dac753d000.ost_server_uuid 50 [ 4276.558682] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92dac753d000.ost_server_uuid in IDLE state after 0 sec [ 4288.210568] Lustre: server umount lustre-OST0001 complete [ 4291.042381] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.046861] LustreError: 141269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.065411] Lustre: Skipped 2 previous similar messages [ 4291.083626] LustreError: 141269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4293.103586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.114933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.137087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.153076] Lustre: Skipped 1 previous similar message [ 4296.170325] LustreError: 140330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.202597] LustreError: 140330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4299.291333] Lustre: server umount lustre-OST0000 complete [ 4304.352368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4304.360390] LustreError: Skipped 1 previous similar message [ 4304.373046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.387603] Lustre: Skipped 1 previous similar message [ 4304.392884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.395698] Lustre: Skipped 3 previous similar messages [ 4309.943859] Lustre: server umount lustre-MDT0000 complete [ 4311.521694] LustreError: 138508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.553387] LustreError: 138508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4313.975885] LustreError: 138489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483940 with bad export cookie 6884427583777789956 [ 4313.977901] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.982840] LustreError: 138489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4314.304748] Lustre: server umount lustre-MDT0001 complete [ 4337.256321] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4340.631777] Key type lgssc unregistered [ 4340.926199] LNet: 144886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4340.931074] LNetError: 144886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4340.944736] LNet: Removed LNI 192.168.201.113@tcp [ 4341.844109] Key type .llcrypt unregistered [ 4341.848102] Key type ._llcrypt unregistered [ 4382.019451] Key type ._llcrypt registered [ 4382.021780] Key type .llcrypt registered [ 4382.109488] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4396.633721] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4397.995116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4398.074261] alg: No test for adler32 (adler32-zlib) [ 4399.238928] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4399.661288] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4401.367443] Key type lgssc registered [ 4402.547491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4440.568574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4440.595394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.871896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4441.893333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4441.958930] Lustre: lustre-MDT0000: new disk, initializing [ 4442.036190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.057557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4447.099713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4458.690398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.817477] Lustre: 149032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4458.864302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4458.868269] Lustre: Skipped 1 previous similar message [ 4458.920647] Lustre: lustre-MDT0001: new disk, initializing [ 4458.997955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4459.071716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4459.087123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4463.746559] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4473.515688] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.525160] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4486.783879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.001423] Lustre: lustre-OST0000: new disk, initializing [ 4487.005145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4487.013688] Lustre: 151000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4487.158558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.938395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4493.949591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4494.094354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4494.317829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4504.423459] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4509.070244] Lustre: Failing over lustre-OST0000 [ 4509.224493] Lustre: server umount lustre-OST0000 complete [ 4525.023311] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484135/real 1789484135] req@00000000d1b9d0c9 x1876410001033472/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789484151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4525.027617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.058152] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4527.136332] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484137/real 1789484137] req@00000000c1aff794 x1876410001033856/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789484153 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4527.169379] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4528.386025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.581345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.606016] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4528.668079] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4531.679396] Lustre: 147254:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484140/real 1789484140] req@0000000098f5b327 x1876410001034624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789484156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4535.391356] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4535.798559] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4535.803913] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4536.865129] Lustre: 147253:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484146/real 1789484146] req@000000009181d231 x1876410001035776/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789484162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4536.887625] Lustre: 147253:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4539.386388] Lustre: Failing over lustre-MDT0000 [ 4539.718954] Lustre: server umount lustre-MDT0000 complete [ 4540.902712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.914872] Lustre: Skipped 1 previous similar message [ 4540.925591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4541.923130] Lustre: 147254:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484151/real 1789484151] req@000000000d526a51 x1876410001036672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789484167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4541.956593] Lustre: 147254:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4546.016807] LustreError: 150256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.053609] LustreError: 150256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4549.061759] LustreError: 149039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.139813] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.158881] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4554.190982] LustreError: 151008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.286209] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789484167/real 1789484167] req@00000000fafccade x1876410001045376/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789484183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4557.331482] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.139048] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.155332] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4559.939362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.541696] LustreError: 149044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.543427] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x934a3541dbb7ab58 [ 4567.559824] LustreError: 149044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4567.578815] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [ 4567.592022] Lustre: Skipped 1 previous similar message [ 4567.950194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.034564] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4569.548846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4573.060653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4573.183442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4573.188121] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4573.251268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4583.411030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.420261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.431601] Lustre: Skipped 2 previous similar messages [ 4583.451379] Lustre: Skipped 1 previous similar message [ 4588.528456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.538857] Lustre: Skipped 1 previous similar message [ 4589.317549] Lustre: server umount lustre-OST0000 complete [ 4593.633617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.634626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.648768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.683985] Lustre: Skipped 1 previous similar message [ 4598.754633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4598.765250] Lustre: Skipped 2 previous similar messages [ 4599.885432] Lustre: server umount lustre-MDT0000 complete [ 4603.873025] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.884440] LustreError: 149038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4604.831987] LustreError: 149024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484230 with bad export cookie 10613354028827388760 [ 4604.838827] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.841672] LustreError: 149024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4605.433909] Lustre: server umount lustre-MDT0001 complete [ 4617.009459] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4620.288868] Key type lgssc unregistered [ 4620.575536] LNet: 154656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4620.582630] LNetError: 154656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4620.603899] LNet: Removed LNI 192.168.201.113@tcp [ 4621.439221] Key type .llcrypt unregistered [ 4621.440932] Key type ._llcrypt unregistered [ 4661.763740] Key type ._llcrypt registered [ 4661.766839] Key type .llcrypt registered [ 4661.879341] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4662.613241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4662.685481] alg: No test for adler32 (adler32-zlib) [ 4664.263324] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4664.627256] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4666.415596] Key type lgssc registered [ 4667.938343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4675.645296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4675.673643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.131341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4682.972847] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4687.331146] LustreError: 156173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.370177] LustreError: 156173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4692.450222] LustreError: 156174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.466326] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.567983] LustreError: 156173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4699.109926] LustreError: 156202:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4699.122573] LustreError: 156202:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4699.127880] LustreError: 156202:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4699.642801] Lustre: server umount lustre-MDT0000 complete [ 4719.137270] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4728.782217] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4774.404495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.988479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4775.049349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4775.221176] Lustre: lustre-MDT0000: new disk, initializing [ 4775.360901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.440387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4781.889944] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4793.724880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.776162] Lustre: 159510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4793.800594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4793.802425] Lustre: Skipped 1 previous similar message [ 4793.855612] Lustre: lustre-MDT0001: new disk, initializing [ 4793.972192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4794.019914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4794.037198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4800.051437] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4811.272622] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4818.228435] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.746315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4826.013543] Lustre: lustre-OST0000: new disk, initializing [ 4826.017944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4826.028281] Lustre: 161481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4826.125262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.471602] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4835.365447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4835.370295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4835.511364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4842.150662] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4847.118776] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4848.754534] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:01:14 (1789484474) [ 5095.903801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5095.907903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.916276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.930621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.950268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.985189] Lustre: Skipped 1 previous similar message [ 5100.047448] Lustre: server umount lustre-MDT0000 complete [ 5102.049640] LustreError: 159521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.074987] LustreError: 159521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5103.929976] LustreError: 160315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484730 with bad export cookie 17273875449734247077 [ 5103.943912] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.968468] LustreError: 160315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5107.170936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.172261] LustreError: 163262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.184534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5107.184544] Lustre: Skipped 2 previous similar messages [ 5110.310245] Lustre: server umount lustre-MDT0001 complete [ 5129.183657] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5129.474944] Lustre: server umount lustre-OST0000 complete [ 5141.683535] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5153.259591] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5199.375714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.711964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5199.745880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5199.830989] Lustre: lustre-MDT0000: new disk, initializing [ 5199.930746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.951488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5206.492177] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5219.758420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.970573] Lustre: 166158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5220.025427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5220.030927] Lustre: Skipped 1 previous similar message [ 5220.146436] Lustre: lustre-MDT0001: new disk, initializing [ 5220.319603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.452039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5220.478459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5226.652683] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5238.374385] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5244.794807] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5253.544265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.899663] Lustre: lustre-OST0000: new disk, initializing [ 5253.905993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5253.918937] Lustre: 168128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5254.004590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.866051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5260.876703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5261.001074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5261.404056] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5271.668267] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.497665] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5277.873568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5284.258657] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5284.545320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5286.883662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5286.905295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.946720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5286.951274] Lustre: Skipped 2 previous similar messages [ 5292.011766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.023787] Lustre: Skipped 1 previous similar message [ 5301.216304] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5301.519936] Lustre: server umount lustre-OST0000 complete [ 5307.360531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5307.363585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.379714] LustreError: Skipped 1 previous similar message [ 5307.382343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.390479] Lustre: Skipped 2 previous similar messages [ 5307.413885] Lustre: Skipped 3 previous similar messages [ 5320.162318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5320.456923] Lustre: server umount lustre-MDT0000 complete [ 5322.721550] LustreError: 168821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.745693] LustreError: 168821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5324.307714] LustreError: 166149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484950 with bad export cookie 17273875449734405907 [ 5324.309208] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5324.323233] LustreError: 166149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5324.676989] Lustre: server umount lustre-MDT0001 complete [ 5335.587885] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:09:20 (1789484960) [ 5353.746769] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5363.989805] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5411.197977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.838951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.967058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5412.117291] Lustre: lustre-MDT0000: new disk, initializing [ 5412.328452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.378382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5420.266872] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5434.048452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.236350] Lustre: 172890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5434.534210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5440.587399] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5452.039795] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5458.455955] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5466.985644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.328457] Lustre: lustre-OST0000: new disk, initializing [ 5467.331506] Lustre: Skipped 1 previous similar message [ 5467.337919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5467.342843] Lustre: Skipped 2 previous similar messages [ 5467.349910] Lustre: 174873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.460626] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5477.073340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5477.086634] Lustre: Skipped 1 previous similar message [ 5477.097886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5477.261333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5488.882573] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5497.830869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.832406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.837469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.844666] Lustre: Skipped 1 previous similar message [ 5497.857477] Lustre: Skipped 5 previous similar messages [ 5502.787142] Lustre: server umount lustre-OST0000 complete [ 5522.403300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5522.581465] Lustre: server umount lustre-MDT0000 complete [ 5523.425326] LustreError: 172899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.437984] LustreError: 172899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5526.471062] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485152 with bad export cookie 17273875449734406789 [ 5526.471678] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5526.478825] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5527.085718] Lustre: server umount lustre-MDT0001 complete [ 5537.154645] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:12:42 (1789485162) [ 5554.919786] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5568.354290] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5626.540046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.879327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5626.912929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5626.981403] Lustre: lustre-MDT0000: new disk, initializing [ 5627.104143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.116979] Lustre: Skipped 2 previous similar messages [ 5627.192973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5635.895971] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5652.573856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.696226] Lustre: 179200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5652.913753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5660.945557] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5674.114390] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5681.848251] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5693.771775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.077285] Lustre: 181180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5702.317748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5702.535146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5703.599927] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5718.419832] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5728.227994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.232734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.243510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.243517] Lustre: Skipped 6 previous similar messages [ 5728.243828] LustreError: Skipped 2 previous similar messages [ 5728.289363] Lustre: Skipped 3 previous similar messages [ 5733.356682] LustreError: 181498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.374713] LustreError: 181498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5733.506381] Lustre: server umount lustre-OST0000 complete [ 5740.012288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5745.855271] Lustre: server umount lustre-MDT0000 complete [ 5751.140561] LustreError: 179193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485377 with bad export cookie 17273875449734407874 [ 5751.153491] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5751.174812] LustreError: 179193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5751.829559] Lustre: server umount lustre-MDT0001 complete [ 5769.087340] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5773.701498] Key type lgssc unregistered [ 5774.197985] LNet: 183039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5774.210215] LNetError: 183039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5774.228948] LNet: Removed LNI 192.168.201.113@tcp [ 5775.744592] Key type .llcrypt unregistered [ 5775.751298] Key type ._llcrypt unregistered [ 5805.779480] Key type ._llcrypt registered [ 5805.781773] Key type .llcrypt registered [ 5805.889833] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5807.773474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5807.853233] alg: No test for adler32 (adler32-zlib) [ 5809.206952] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 5809.664025] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5811.387097] Key type lgssc registered [ 5813.063039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.426613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5823.484223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.354391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.542686] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5835.750627] LustreError: 184457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.778160] LustreError: 184457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5840.870206] LustreError: 184456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.548594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.161982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.111564] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5859.966755] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5867.011368] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5874.691590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5875.239784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.739917] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5894.776388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5895.744619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5916.136593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.138774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.149735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.253366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.267039] Lustre: Skipped 2 previous similar messages [ 5922.145105] Lustre: server umount lustre-OST0000 complete [ 5930.977303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5930.987092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.016134] Lustre: Skipped 1 previous similar message [ 5931.036043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.889931] Lustre: server umount lustre-MDT0000 complete [ 5936.612451] LustreError: 184451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.636651] LustreError: 184451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5938.512836] LustreError: 184436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485564 with bad export cookie 13514570875339657597 [ 5938.534112] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.550348] LustreError: 184436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5939.142872] Lustre: server umount lustre-MDT0001 complete [ 5953.428388] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5957.263490] Key type lgssc unregistered [ 5957.545322] LNet: 188013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5957.556903] LNetError: 188013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5957.569880] LNet: Removed LNI 192.168.201.113@tcp [ 5959.082867] Key type .llcrypt unregistered [ 5959.088218] Key type ._llcrypt unregistered [ 5993.170419] Key type ._llcrypt registered [ 5993.193909] Key type .llcrypt registered [ 5993.591587] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5994.461426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5994.518373] alg: No test for adler32 (adler32-zlib) [ 5995.529085] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 5995.778955] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5997.495570] Key type lgssc registered [ 5999.553727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6011.625207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6011.637767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.181241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.020753] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6023.654957] LustreError: 189429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.682250] LustreError: 189429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6028.770885] LustreError: 189430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.803664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.326198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6036.296119] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6045.606477] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6053.345933] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6062.244155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.822485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.554647] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6081.929906] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6083.349355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6090.722155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.742611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.784255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.801774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.802214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.830979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.822828] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 6094.828723] Lustre: Skipped 1 previous similar message [ 6096.320609] Lustre: server umount lustre-OST0000 complete [ 6106.785079] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92dac8520000.ost_server_uuid 50 [ 6109.188399] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6127.584892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6127.600740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.622269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.634166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.351130] Lustre: server umount lustre-MDT0000 complete [ 6134.756431] LustreError: 189430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.798247] LustreError: 189430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6136.334353] LustreError: 189409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485762 with bad export cookie 6341493777997176426 [ 6136.335356] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.348285] LustreError: 189409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6136.724754] Lustre: server umount lustre-MDT0001 complete [ 6149.727545] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:22:54 (1789485774) [ 6160.439790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.005097] LustreError: 193352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.135553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.498151] LustreError: 193352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.515686] LustreError: 193352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6167.329268] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6176.740348] LustreError: 193353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.762393] LustreError: 193353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6178.122590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.702787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6185.099267] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6194.235562] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.570772] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.861531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6213.482746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.027352] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6230.909410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6232.782597] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.573930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.932339] Lustre: lustre-OST0001: new disk, initializing [ 6243.937117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6243.948853] Lustre: 196198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6244.112439] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6250.823723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6250.844042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6251.158531] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6254.275918] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6267.299749] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6271.969084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.985818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.015476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.029647] Lustre: Skipped 2 previous similar messages [ 6277.888629] Lustre: server umount lustre-OST0000 complete [ 6278.618158] LustreError: 195175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.824730] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6284.994216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6290.827563] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6291.226724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6307.304775] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.323378] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.344261] LustreError: Skipped 2 previous similar messages [ 6307.373773] Lustre: Skipped 1 previous similar message [ 6307.391869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6307.433202] Lustre: Skipped 5 previous similar messages [ 6312.376778] Lustre: server umount lustre-OST0001 complete [ 6317.024994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6327.777381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.785421] Lustre: Skipped 5 previous similar messages [ 6331.360538] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6331.766183] Lustre: server umount lustre-MDT0000 complete [ 6332.907851] LustreError: 193348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.932683] LustreError: 193348:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6336.597354] LustreError: 195197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485962 with bad export cookie 6341493777997177497 [ 6336.608763] LustreError: 195197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6336.616580] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.015482] Lustre: server umount lustre-MDT0001 complete [ 6368.220765] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:26:33 (1789485993) [ 6377.454288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.839571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6377.876239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6378.085458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.787978] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6397.593858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.759099] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6397.839858] Lustre: 199641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6407.217993] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6419.257721] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6430.106570] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6440.549632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6441.087478] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6447.301817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6448.252326] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6457.554398] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.186931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6466.249107] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6466.684416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.692803] Lustre: Skipped 2 previous similar messages [ 6474.162282] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6483.466725] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6487.528243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.535786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6487.566896] Lustre: Skipped 4 previous similar messages [ 6487.586562] Lustre: Skipped 1 previous similar message [ 6493.156814] Lustre: server umount lustre-OST0000 complete [ 6494.151112] LustreError: 201496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.123320] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6513.151775] LustreError: Skipped 1 previous similar message [ 6516.054726] Lustre: server umount lustre-OST0001 complete [ 6521.312848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.332466] Lustre: Skipped 2 previous similar messages [ 6527.879838] Lustre: server umount lustre-MDT0000 complete [ 6532.088817] LustreError: 199632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486158 with bad export cookie 6341493777997178904 [ 6532.096048] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6532.104241] LustreError: 199632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6532.658484] Lustre: server umount lustre-MDT0001 complete [ 6560.393539] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:29:45 (1789486185) [ 6579.696154] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 6590.540234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6631.170147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.529120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6631.581119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6631.716107] Lustre: lustre-MDT0000: new disk, initializing [ 6631.801748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6631.838527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6636.862701] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6647.845375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.022367] Lustre: 207745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6648.137784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6648.143082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6648.149322] Lustre: Skipped 1 previous similar message [ 6648.158941] Lustre: Skipped 8 previous similar messages [ 6648.228984] Lustre: lustre-MDT0001: new disk, initializing [ 6648.341551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6648.351755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6653.218722] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6663.142825] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6670.326417] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6677.829501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6678.155388] Lustre: lustre-OST0000: new disk, initializing [ 6678.162290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6678.170530] Lustre: 209714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6685.287816] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6687.791410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6687.806703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6687.948513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6695.141416] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6700.138844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6700.320315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6704.453854] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6704.720707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6708.704204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6708.710989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.711017] Lustre: Skipped 1 previous similar message [ 6708.754427] LustreError: Skipped 1 previous similar message [ 6720.992599] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6721.207565] Lustre: server umount lustre-OST0000 complete [ 6733.519039] Lustre: server umount lustre-MDT0000 complete [ 6734.306474] LustreError: 207751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.328502] LustreError: 207751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6737.432670] LustreError: 207735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486363 with bad export cookie 6341493777997180465 [ 6737.435707] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.453345] LustreError: 207735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6738.034133] Lustre: server umount lustre-MDT0001 complete [ 6747.875166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.901824] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6762.486451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.574879] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6776.468217] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6782.854897] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6790.474574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.301863] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6806.206313] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6810.774317] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6811.035575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6815.451188] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6815.800414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6819.809353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6819.816619] LustreError: Skipped 1 previous similar message [ 6819.821814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.830455] Lustre: Skipped 9 previous similar messages [ 6824.360034] Lustre: server umount lustre-OST0000 complete [ 6833.111670] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6840.119373] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6869.869647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.257174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6873.268881] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6876.701956] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6884.851714] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.545885] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6888.622237] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6888.662035] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6893.631389] Lustre: server umount lustre-OST0000 complete [ 6899.169811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.203357] Lustre: Skipped 6 previous similar messages [ 6908.473676] Lustre: server umount lustre-MDT0000 complete [ 6910.303408] Lustre: 188737:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789486520/real 1789486520] req@000000008a0a6c38 x1876411674802688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789486536 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6912.408425] LustreError: 211912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486538 with bad export cookie 6341493777997181347 [ 6912.421420] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.963584] Lustre: server umount lustre-MDT0001 complete [ 6924.607805] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:35:49 (1789486549) [ 6932.695387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.186131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.191429] Lustre: Skipped 6 previous similar messages [ 6937.621708] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6948.217435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.090884] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6964.207989] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6970.778643] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6978.237283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.409705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6991.920463] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6996.399884] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6996.637394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7001.334740] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7001.651692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.610396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7008.798419] Lustre: lustre-OST0001: new disk, initializing [ 7008.805309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7008.814221] Lustre: 221273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7016.363192] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7017.043571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7017.052586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7017.154721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7026.710904] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.545390] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7032.770929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7037.710429] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7037.963926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7043.040616] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.060872] LustreError: Skipped 1 previous similar message [ 7054.303490] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7054.436813] Lustre: server umount lustre-OST0001 complete [ 7058.414943] LustreError: 221670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.453671] LustreError: 221670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7062.626884] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7062.805482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7068.401074] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7068.844508] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7098.774041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7104.511134] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7104.535535] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7104.560092] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7106.887910] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7115.167315] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7116.704596] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7119.854868] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7119.863502] Lustre: Skipped 12 previous similar messages [ 7125.051550] Lustre: server umount lustre-OST0001 complete [ 7149.379849] Lustre: server umount lustre-OST0000 complete [ 7155.680333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.707322] Lustre: Skipped 7 previous similar messages [ 7160.224532] Lustre: server umount lustre-MDT0000 complete [ 7164.389649] LustreError: 217885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486790 with bad export cookie 6341493777997182313 [ 7164.395192] LustreError: 217885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7164.398312] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.881192] Lustre: server umount lustre-MDT0001 complete [ 7196.679860] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:40:22 (1789486822) [ 7203.162257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.403993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7203.435948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7208.111725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7216.892950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.021534] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7217.075484] Lustre: 226522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7222.398869] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7230.264516] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.430637] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.673928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.953720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7252.092255] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7260.471085] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.215511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.281719] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7277.304734] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7286.041950] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7288.094424] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7293.120485] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7293.301355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7298.679966] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7298.955432] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7305.515324] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92dac38f9800.ost_server_uuid 50 [ 7307.259726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92dac38f9800.ost_server_uuid in IDLE state after 0 sec [ 7308.634384] Lustre: Permanently deactivating lustre-OST0001 [ 7308.648885] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7313.212778] Lustre: Permanently reactivating lustre-OST0001 [ 7313.237727] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7313.243819] Lustre: Skipped 2 previous similar messages [ 7319.013921] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7319.024142] LustreError: Skipped 2 previous similar messages [ 7333.344660] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7333.485529] Lustre: server umount lustre-OST0001 complete [ 7343.912764] Lustre: server umount lustre-OST0000 complete [ 7354.627420] Lustre: server umount lustre-MDT0000 complete [ 7358.495314] LustreError: 226517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486984 with bad export cookie 6341493777997183594 [ 7358.501756] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.521243] LustreError: 226517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7358.914809] Lustre: server umount lustre-MDT0001 complete [ 7387.965907] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:43:32 (1789487012) [ 7396.198896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.565403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7396.644309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7396.668216] Lustre: Skipped 2 previous similar messages [ 7402.084703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7411.361281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.483148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7411.505565] Lustre: 233224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7416.525160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7425.122203] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7432.679692] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7440.826923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7441.178951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7441.221083] Lustre: Permanently deactivating lustre-OST0000 [ 7441.234886] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7448.713948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7451.688556] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7456.163719] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.826398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7463.892990] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7464.123059] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7464.131102] Lustre: Skipped 11 previous similar messages [ 7469.877743] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7471.241706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7478.478869] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7480.559238] Lustre: Permanently reactivating lustre-OST0000 [ 7480.585651] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7480.594886] Lustre: Skipped 2 previous similar messages [ 7480.847189] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7480.852753] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7480.885361] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7480.896746] Lustre: Skipped 1 previous similar message [ 7480.969823] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7480.992874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7500.525375] Lustre: server umount lustre-OST0001 complete [ 7511.449096] Lustre: server umount lustre-OST0000 complete [ 7521.952415] Lustre: server umount lustre-MDT0000 complete [ 7525.279183] LustreError: 233214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487151 with bad export cookie 6341493777997185267 [ 7525.283242] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.291548] LustreError: 233214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7525.702793] Lustre: server umount lustre-MDT0001 complete [ 7537.133774] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7540.530841] Key type lgssc unregistered [ 7540.808499] LNet: 238099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.817571] LNetError: 238099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7540.825740] LNet: Removed LNI 192.168.201.113@tcp [ 7541.607445] Key type .llcrypt unregistered [ 7541.609192] Key type ._llcrypt unregistered [ 7566.458257] Key type ._llcrypt registered [ 7566.466685] Key type .llcrypt registered [ 7566.577598] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7567.957200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.131858] alg: No test for adler32 (adler32-zlib) [ 7569.287401] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 7569.603309] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7571.343240] Key type lgssc registered [ 7572.772257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.597467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.624913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.961815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.827238] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7595.492403] LustreError: 239564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.513014] LustreError: 239564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7598.837837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.985080] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7599.016331] Lustre: 239550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7599.023676] Lustre: Permanently deactivating lustre-MDT0001 [ 7599.039516] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7599.347397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7604.051674] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7604.798787] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7613.058211] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7619.517391] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7627.129982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.512514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.587859] LustreError: 241583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.604553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7635.812257] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7636.965286] LustreError: 241407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.086756] LustreError: 241405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.174383] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7644.537197] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7649.257787] LustreError: 241406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7649.273573] LustreError: 241406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7652.177598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.542462] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.979381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7658.000736] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7659.935666] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7669.417497] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7672.904991] Lustre: Permanently reactivating lustre-MDT0001 [ 7672.942992] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7672.951960] Lustre: Skipped 2 previous similar messages [ 7677.934403] Lustre: lustre-MDT0001: Client 1db7738f-0ff4-44da-b0d4-716cd9cb1fc6 (at 192.168.201.13@tcp) reconnecting [ 7683.229550] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.244398] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7683.249952] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7683.259383] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7683.260568] LustreError: 243278:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7683.290631] LustreError: 243278:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7683.298850] LustreError: 243278:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -108 [ 7683.313066] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7683.313066] [ 7683.325364] LustreError: 239555:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7685.262688] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7687.679909] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7693.929844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7694.190463] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7699.014884] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7699.287681] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7702.546366] Lustre: Permanently deactivating lustre-MDT0001 [ 7702.567312] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7702.572711] Lustre: Skipped 2 previous similar messages [ 7708.777776] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7717.344205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7717.350093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.359333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.395438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.399380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.418320] Lustre: Skipped 2 previous similar messages [ 7719.434482] Lustre: Skipped 2 previous similar messages [ 7723.100117] Lustre: server umount lustre-MDT0000 complete [ 7724.512569] LustreError: 240974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.541102] LustreError: 240974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7726.820847] LustreError: 244051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487352 with bad export cookie 11000796800527762048 [ 7726.822836] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7726.827046] LustreError: 244051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7727.173552] Lustre: server umount lustre-MDT0001 complete [ 7733.437882] Lustre: server umount lustre-OST0000 complete [ 7749.333560] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:49:33 (1789487373) [ 7764.670963] Lustre: server umount lustre-OST0001 complete [ 7771.296583] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 7781.817987] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7825.665620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.158181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7826.170100] Lustre: Skipped 2 previous similar messages [ 7826.203309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7826.314716] Lustre: lustre-MDT0000: new disk, initializing [ 7826.467302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.491906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7831.814969] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7845.410804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7845.589427] Lustre: 247828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7845.672647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7845.677394] Lustre: Skipped 1 previous similar message [ 7845.849127] Lustre: lustre-MDT0001: new disk, initializing [ 7845.967993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7846.025768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7846.035793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7850.758883] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7860.302375] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7867.092635] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7875.276952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.521735] Lustre: lustre-OST0000: new disk, initializing [ 7875.525386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7875.532820] Lustre: 249800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7875.633645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.797905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7880.807326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7880.956787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7882.940815] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7893.410222] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7895.780404] LustreError: 247834:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7905.563650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7905.822896] Lustre: lustre-OST0001: new disk, initializing [ 7905.823779] LustreError: 247834:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7905.826493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7905.836309] Lustre: 250913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7905.863143] LustreError: 249516:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7911.565837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7911.572933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7911.664423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7913.008504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7915.943174] LustreError: 249516:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7923.942385] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7926.002802] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7932.408178] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7932.415557] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.423817] LustreError: Skipped 1 previous similar message [ 7932.460631] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7932.480713] Lustre: Skipped 2 previous similar messages [ 7934.550804] Lustre: server umount lustre-OST0001 complete [ 7937.524822] LustreError: 249783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7937.556904] LustreError: 249783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7957.991901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7957.995709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.010216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7958.026126] Lustre: Skipped 2 previous similar messages [ 7959.635108] Lustre: server umount lustre-OST0000 complete [ 7968.224330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.228603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7968.248346] Lustre: Skipped 1 previous similar message [ 7968.254638] Lustre: Skipped 1 previous similar message [ 7970.862710] Lustre: server umount lustre-MDT0000 complete [ 7973.347311] LustreError: 247838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.385754] LustreError: 247838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7975.683349] LustreError: 247819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487601 with bad export cookie 11000796800527764862 [ 7975.694687] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.696457] LustreError: 247819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7982.218382] Lustre: server umount lustre-MDT0001 complete [ 7994.910597] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7998.096032] Key type lgssc unregistered [ 7998.490196] LNet: 252949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7998.502954] LNetError: 252949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7998.531446] LNet: Removed LNI 192.168.201.113@tcp [ 7999.403987] Key type .llcrypt unregistered [ 7999.405696] Key type ._llcrypt unregistered [ 8042.602196] Key type ._llcrypt registered [ 8042.608492] Key type .llcrypt registered [ 8042.741893] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8043.783818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8043.910368] alg: No test for adler32 (adler32-zlib) [ 8045.512097] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 8045.756179] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8047.434277] Key type lgssc registered [ 8048.688706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.399700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.429093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.775412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8058.816416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8059.045628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.240825] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8074.034374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8074.129294] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8074.157602] Lustre: 255153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8074.425501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8079.360442] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8087.399051] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8093.615388] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8100.708226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.888610] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8101.069381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8108.060267] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8109.223915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8115.634530] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8170.464751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8170.489457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.533023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8170.997625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.007550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.039393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8175.747235] Lustre: server umount lustre-OST0000 complete [ 8181.021658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8197.158433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8197.733663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8205.737509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8211.065543] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8211.082609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8211.085369] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8211.109205] Lustre: Skipped 1 previous similar message [ 8214.980793] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8257.507135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.507620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8257.510499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.514514] Lustre: Skipped 1 previous similar message [ 8257.531306] LustreError: Skipped 1 previous similar message [ 8261.640407] Lustre: server umount lustre-OST0000 complete [ 8267.744984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.759620] Lustre: Skipped 1 previous similar message [ 8267.760303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8267.767232] Lustre: Skipped 2 previous similar messages [ 8271.412103] Lustre: server umount lustre-MDT0000 complete [ 8275.186182] LustreError: 255145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487901 with bad export cookie 7156639032202589215 [ 8275.190598] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.196399] LustreError: 255145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8275.581503] Lustre: server umount lustre-MDT0001 complete [ 8287.765869] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8291.058635] Key type lgssc unregistered [ 8291.462164] LNet: 260579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8291.484830] LNetError: 260579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8292.518899] LNet: Removed LNI 192.168.201.113@tcp [ 8293.603320] Key type .llcrypt unregistered [ 8293.605570] Key type ._llcrypt unregistered [ 8323.558534] Key type ._llcrypt registered [ 8323.564597] Key type .llcrypt registered [ 8323.724231] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8325.305894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8325.354803] alg: No test for adler32 (adler32-zlib) [ 8326.416761] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 8326.717067] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8328.351519] Key type lgssc registered [ 8329.540410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8337.581466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8337.596793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.118826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.554100] LustreError: 261995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.593209] LustreError: 261995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8345.443747] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8349.688933] LustreError: 261994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8354.790051] LustreError: 261995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.971953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.377552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8361.027476] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8369.284384] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8376.601956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8384.151866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.513255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.976555] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8400.672682] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8409.087801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8437.735912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.754373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.772728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8439.779084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8439.788838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.791480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.761607] Lustre: server umount lustre-OST0000 complete [ 8448.480359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8448.485969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.498045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8448.502058] Lustre: Skipped 1 previous similar message [ 8452.844618] Lustre: server umount lustre-MDT0000 complete [ 8455.136431] LustreError: 261989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.159358] LustreError: 261989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8456.645545] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488082 with bad export cookie 3548491355388627393 [ 8456.660706] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8456.666591] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8457.114543] Lustre: server umount lustre-MDT0001 complete [ 8470.872677] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8474.116770] Key type lgssc unregistered [ 8474.440843] LNet: 266334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8474.445151] LNetError: 266334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8474.467202] LNet: Removed LNI 192.168.201.113@tcp [ 8475.392874] Key type .llcrypt unregistered [ 8475.398270] Key type ._llcrypt unregistered [ 8494.729913] Key type ._llcrypt registered [ 8494.734912] Key type .llcrypt registered [ 8494.851073] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8496.023661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.143564] alg: No test for adler32 (adler32-zlib) [ 8497.324033] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 8497.560401] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8499.241640] Key type lgssc registered [ 8500.347213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.214306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.232865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.955196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.521978] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8521.186879] LustreError: 267553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.203956] LustreError: 267553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8526.307327] LustreError: 267552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.033611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.580456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8535.428491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8543.288899] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.842458] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8555.409356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8555.883386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.940933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8564.203599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8573.638345] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.749342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8587.761306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.784610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8589.792811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8589.804815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.819242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8589.822664] Lustre: Skipped 1 previous similar message [ 8592.921893] Lustre: server umount lustre-OST0000 complete [ 8600.035727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.037407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8600.053115] Lustre: Skipped 1 previous similar message [ 8600.070351] Lustre: Skipped 1 previous similar message [ 8603.856730] Lustre: server umount lustre-MDT0000 complete [ 8605.159732] LustreError: 267553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8605.175580] LustreError: 267553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8609.115537] LustreError: 267534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488235 with bad export cookie 4687717908340755690 [ 8609.133058] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8609.150502] LustreError: 267534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8609.687578] Lustre: server umount lustre-MDT0001 complete [ 8622.199308] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8625.304121] Key type lgssc unregistered [ 8625.667795] LNet: 271231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8625.708328] LNetError: 271231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8625.763187] LNet: Removed LNI 192.168.201.113@tcp [ 8626.771507] Key type .llcrypt unregistered [ 8626.778140] Key type ._llcrypt unregistered [ 8657.529209] Key type ._llcrypt registered [ 8657.531278] Key type .llcrypt registered [ 8657.629521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8659.155429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8659.220146] alg: No test for adler32 (adler32-zlib) [ 8660.393036] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 8660.756165] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8662.471313] Key type lgssc registered [ 8663.563655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8672.836376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8672.868901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.418848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.157666] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8684.016384] LustreError: 272648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.060019] LustreError: 272648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8689.127319] LustreError: 272647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8692.450450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.717806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8698.523913] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8709.539761] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8718.335748] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8726.093372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8726.350991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.243164] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8739.738494] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8744.769857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8773.600514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.618688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.668798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.649899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.669435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.695897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8778.696709] Lustre: server umount lustre-OST0000 complete [ 8784.864890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8784.882450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.904252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.907578] Lustre: Skipped 1 previous similar message [ 8789.571182] Lustre: server umount lustre-MDT0000 complete [ 8791.009104] LustreError: 273376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.020416] LustreError: 273376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8793.364642] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488419 with bad export cookie 11822430440499202966 [ 8793.374649] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8793.375641] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8793.693367] Lustre: server umount lustre-MDT0001 complete [ 8806.020233] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8809.070545] Key type lgssc unregistered [ 8809.418497] LNet: 276955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8809.430935] LNetError: 276955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8809.480563] LNet: Removed LNI 192.168.201.113@tcp [ 8810.440619] Key type .llcrypt unregistered [ 8810.442753] Key type ._llcrypt unregistered [ 8829.498516] Key type ._llcrypt registered [ 8829.503110] Key type .llcrypt registered [ 8829.656785] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8831.165799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8831.482375] alg: No test for adler32 (adler32-zlib) [ 8832.601590] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 8832.927426] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8834.632365] Key type lgssc registered [ 8836.238888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8845.471771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8845.490977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8847.030589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.826614] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8857.576767] LustreError: 278190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.602311] LustreError: 278190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8861.189035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.518918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.127459] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8874.756240] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8880.670733] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8888.548060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.015928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.098207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8896.454575] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8906.041865] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8918.316729] Lustre: server umount lustre-OST0000 complete [ 8923.107712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8923.112786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.122408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.208664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.209654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.321832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.329791] Lustre: Skipped 2 previous similar messages [ 8933.534716] Lustre: server umount lustre-MDT0000 complete [ 8937.453733] LustreError: 278171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.482152] LustreError: 278171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8937.853638] LustreError: 278156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488564 with bad export cookie 2058495075412411957 [ 8937.859192] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8937.867966] LustreError: 278156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8948.448419] Lustre: server umount lustre-MDT0001 complete [ 8962.000508] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8965.982433] Key type lgssc unregistered [ 8966.399774] LNet: 281894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8966.415601] LNetError: 281894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8966.451234] LNet: Removed LNI 192.168.201.113@tcp [ 8968.480105] Key type .llcrypt unregistered [ 8968.481781] Key type ._llcrypt unregistered [ 8987.158145] Key type ._llcrypt registered [ 8987.159986] Key type .llcrypt registered [ 8987.316518] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9007.546641] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9026.770962] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9027.823666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9027.881223] alg: No test for adler32 (adler32-zlib) [ 9029.237412] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 9029.560825] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9031.322581] Key type lgssc registered [ 9032.861662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9075.034431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9075.061970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.566423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9076.610913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9076.722935] Lustre: lustre-MDT0000: new disk, initializing [ 9076.835299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.875527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9082.214841] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9094.320692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.583499] Lustre: 285456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9094.679717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9094.686482] Lustre: Skipped 1 previous similar message [ 9094.825893] Lustre: lustre-MDT0001: new disk, initializing [ 9094.996371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9095.069844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9095.082850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9100.792966] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9112.234421] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9118.911520] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9127.488396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9127.927840] Lustre: lustre-OST0000: new disk, initializing [ 9127.930402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9127.942428] Lustre: 287427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9128.089681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9134.809657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9134.829221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9134.996876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9136.256837] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9147.389156] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9152.576456] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9152.878444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9158.904911] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9159.086965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9165.793553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9165.798644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.825082] Lustre: Skipped 1 previous similar message [ 9165.828113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.834517] Lustre: Skipped 1 previous similar message [ 9167.568220] Lustre: server umount lustre-OST0000 complete [ 9171.936683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9171.943589] LustreError: Skipped 1 previous similar message [ 9171.950691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.967705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.032619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.032974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.082497] Lustre: Skipped 1 previous similar message [ 9181.153179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.167580] Lustre: Skipped 1 previous similar message [ 9186.271872] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9186.280951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9186.295719] Lustre: Skipped 1 previous similar message [ 9186.639929] Lustre: server umount lustre-MDT0000 complete [ 9191.609565] LustreError: 285450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488817 with bad export cookie 2668853086245096332 [ 9191.618599] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9191.621278] LustreError: 285450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9192.296290] Lustre: server umount lustre-MDT0001 complete [ 9203.405163] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:13:48 (1789488828) [ 9211.524916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.592718] LustreError: 289843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.626265] LustreError: 289843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9212.843169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.043211] LustreError: 289844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.810652] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9222.183962] LustreError: 289843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.233330] LustreError: 289844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.449628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.879346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9236.351767] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9245.098484] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9252.209718] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9259.940496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9260.498980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9267.845517] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9275.967611] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.377291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9297.381751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.413160] Lustre: Skipped 1 previous similar message [ 9297.414170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9297.433177] Lustre: Skipped 1 previous similar message [ 9302.523548] LustreError: 291667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.798694] Lustre: server umount lustre-OST0000 complete [ 9307.622287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.961781] Lustre: server umount lustre-MDT0000 complete [ 9312.748094] LustreError: 290791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9312.785624] LustreError: 290791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9312.980099] LustreError: 289825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488939 with bad export cookie 2668853086245097214 [ 9312.991200] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9312.995476] LustreError: 289825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9313.399779] Lustre: server umount lustre-MDT0001 complete [ 9327.294542] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9331.054157] Key type lgssc unregistered [ 9331.408859] LNet: 293406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9331.419946] LNetError: 293406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9331.453167] LNet: Removed LNI 192.168.201.113@tcp [ 9332.512191] Key type .llcrypt unregistered [ 9332.514956] Key type ._llcrypt unregistered [ 9362.140351] Key type ._llcrypt registered [ 9362.143514] Key type .llcrypt registered [ 9362.350599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9380.907551] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9383.127964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9383.164468] alg: No test for adler32 (adler32-zlib) [ 9384.483050] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 9384.802379] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9386.527341] Key type lgssc registered [ 9388.221805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9447.563586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9447.634536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.430296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.484326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9449.829497] Lustre: lustre-MDT0000: new disk, initializing [ 9450.088930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.132630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9458.397571] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9473.177915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9473.409427] Lustre: 297229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9473.518922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9473.539948] Lustre: Skipped 1 previous similar message [ 9473.699564] Lustre: lustre-MDT0001: new disk, initializing [ 9473.862265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9473.946562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9473.962823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9479.926167] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9490.818497] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9498.105664] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9506.405642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9506.873518] Lustre: lustre-OST2710: new disk, initializing [ 9506.887523] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9506.912396] Lustre: 299200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9507.092493] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9515.377485] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9515.604821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9515.616239] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9515.906207] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9526.775893] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9534.866130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9535.051505] Lustre: lustre-OST03e8: new disk, initializing [ 9535.055087] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9535.061350] Lustre: 300264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9535.169843] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9542.819586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9542.831758] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9543.012423] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9543.043497] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9553.529605] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9561.408304] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9561.723736] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9566.670986] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9566.969319] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9571.988548] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9572.297620] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9578.161495] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9578.358579] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9586.657504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9586.666126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9586.710405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.220665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.228192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.243214] Lustre: Skipped 2 previous similar messages [ 9590.784509] Lustre: server umount lustre-MDT0000 complete [ 9595.672492] LustreError: 297222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489221 with bad export cookie 8823868953126810376 [ 9595.677363] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9595.679684] LustreError: 297222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9596.061141] Lustre: server umount lustre-MDT0001 complete [ 9611.352217] Lustre: server umount lustre-OST2710 complete [ 9625.453418] Lustre: server umount lustre-OST03e8 complete [ 9653.155898] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9665.159315] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9710.471445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.951527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9711.012241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9711.141150] Lustre: lustre-MDT0000: new disk, initializing [ 9711.256933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.275384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9716.568959] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9729.037314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.209873] Lustre: 305523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9729.266096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9729.269896] Lustre: Skipped 1 previous similar message [ 9729.351779] Lustre: lustre-MDT0001: new disk, initializing [ 9729.451540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9729.497826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9729.507147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9734.693247] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9745.557894] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9753.297634] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.654881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.866962] Lustre: lustre-OST0000: new disk, initializing [ 9763.869074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9763.871516] Lustre: 307490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.973604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.091293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9771.110387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9771.249931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9772.087546] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9783.388926] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9788.649582] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9788.939635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9795.092751] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9795.299714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9802.233989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.234156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.234831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.234836] Lustre: Skipped 3 previous similar messages [ 9802.249920] LustreError: Skipped 1 previous similar message [ 9802.306939] Lustre: Skipped 1 previous similar message [ 9804.016733] Lustre: server umount lustre-OST0000 complete [ 9811.424426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9811.430096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.451536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.458248] Lustre: Skipped 1 previous similar message [ 9817.569401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9817.575043] Lustre: Skipped 3 previous similar messages [ 9825.247921] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9825.389418] Lustre: server umount lustre-MDT0000 complete [ 9827.810625] LustreError: 305535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.827899] LustreError: 305535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9829.857886] LustreError: 305515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489456 with bad export cookie 8823868953126812070 [ 9829.861941] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.884533] LustreError: 305515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9830.464473] Lustre: server umount lustre-MDT0001 complete [ 9840.801038] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:24:26 (1789489466) [ 9842.591748] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9844.855651] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:24:29 (1789489469) [ 9849.666374] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9880.773776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.971563] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9880.987861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9881.097196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.538959] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9895.517922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.671252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9895.759572] Lustre: 311352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9901.599526] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9923.777038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9924.225802] Lustre: Denying initial registration attempt from nid 192.168.201.113@tcp, specified as failover [ 9924.242510] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9924.251452] LustreError: 312808:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9924.260932] LustreError: 312808:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9924.263027] LustreError: 312808:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9924.359819] Lustre: server umount lustre-OST0000 complete [ 9924.367549] LustreError: 312808:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9936.864919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9936.880767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.917493] Lustre: Skipped 1 previous similar message [ 9936.925946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.930959] Lustre: Skipped 2 previous similar messages [ 9943.028779] Lustre: server umount lustre-MDT0000 complete [ 9944.549859] LustreError: 312429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.615053] LustreError: 312429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9949.684726] LustreError: 311363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.730475] LustreError: 311363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9950.561756] LustreError: 311342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489576 with bad export cookie 8823868953126812952 [ 9950.586365] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.592507] LustreError: 311342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9951.042688] Lustre: server umount lustre-MDT0001 complete [ 9964.701612] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9967.996889] Key type lgssc unregistered [ 9968.322397] LNet: 313930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.328372] LNetError: 313930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.350226] LNet: Removed LNI 192.168.201.113@tcp [ 9969.498734] Key type .llcrypt unregistered [ 9969.503879] Key type ._llcrypt unregistered [ 9983.362477] Key type ._llcrypt registered [ 9983.364497] Key type .llcrypt registered [ 9983.500747] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9984.776353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9984.836195] alg: No test for adler32 (adler32-zlib) [ 9986.086713] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 9986.293104] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9987.991197] Key type lgssc registered [ 9989.172719] Lustre: Echo OBD driver; http://www.lustre.org/ [10048.135368] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10057.225482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10057.234703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.461626] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10058.487747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10058.609817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10064.312795] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10074.798483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.918164] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10074.943620] Lustre: 316694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10075.175447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10079.734941] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10090.518050] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10097.650238] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10119.851782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.258370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10120.547768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10128.630413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10138.158310] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10142.179083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10142.198115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10142.202977] LustreError: Skipped 1 previous similar message [10142.205972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10142.217170] Lustre: Skipped 1 previous similar message [10142.247500] Lustre: Skipped 1 previous similar message [10147.299882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10147.311711] Lustre: Skipped 1 previous similar message [10148.588251] Lustre: server umount lustre-OST0000 complete [10157.024540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10157.042919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.064643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.788672] Lustre: server umount lustre-MDT0000 complete [10165.163526] LustreError: 316685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489791 with bad export cookie 16023849663802211277 [10165.175575] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10165.179560] LustreError: 316685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10165.789493] Lustre: server umount lustre-MDT0001 complete [10179.833710] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10183.852817] Key type lgssc unregistered [10184.265741] LNet: 320270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10184.271247] LNetError: 320270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10184.302849] LNet: Removed LNI 192.168.201.113@tcp [10186.035152] Key type .llcrypt unregistered [10186.037196] Key type ._llcrypt unregistered [10215.624936] Key type ._llcrypt registered [10215.631619] Key type .llcrypt registered [10215.865818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10217.480556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10217.519377] alg: No test for adler32 (adler32-zlib) [10219.285862] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [10219.737907] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10221.583245] Key type lgssc registered [10223.170947] Lustre: Echo OBD driver; http://www.lustre.org/ [10233.043681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10233.059739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.456694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10239.141460] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10244.580847] LustreError: 321683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10244.602095] LustreError: 321683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10249.734912] LustreError: 321682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.110534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.714027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10256.540159] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10265.406637] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10273.133915] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10284.731693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.391812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10293.738899] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10302.012193] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10317.281901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10317.293472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.320504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10318.812731] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10320.353507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.354964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.373044] Lustre: Skipped 1 previous similar message [10320.385022] Lustre: Skipped 2 previous similar messages [10322.934838] Lustre: server umount lustre-MDT0000 complete [10323.905534] LustreError: 321679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.349475] LustreError: 321663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789489953 with bad export cookie 18244892297822221372 [10327.374476] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10327.383351] LustreError: 321663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10329.024627] LustreError: 321677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10329.041910] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [10329.063327] LustreError: 321677:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10330.602078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.015462] Lustre: server umount lustre-MDT0001 complete [10343.959683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10358.803670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.106511] LustreError: 323517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.140599] LustreError: 323517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10373.359205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10373.450412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10379.275486] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10382.822726] LustreError: 323518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.843530] LustreError: 323518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10386.957716] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10390.757975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.743988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10397.160593] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10397.323844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10406.528462] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10409.241492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10418.602681] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10421.560592] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10427.880800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.930068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.932083] Lustre: Skipped 2 previous similar messages [10432.483360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10432.488273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10433.858815] Lustre: server umount lustre-OST0000 complete [10442.726454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10442.753548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10442.782930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10442.790744] Lustre: Skipped 2 previous similar messages [10445.179750] Lustre: server umount lustre-MDT0000 complete [10448.353941] LustreError: 326603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.382596] LustreError: 326603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10450.043878] LustreError: 324495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490076 with bad export cookie 18244892297822229205 [10450.046302] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10450.059755] LustreError: 324495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10450.687266] Lustre: server umount lustre-MDT0001 complete [10464.711172] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10468.142418] Key type lgssc unregistered [10468.437461] LNet: 327982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10468.447438] LNetError: 327982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10468.470934] LNet: Removed LNI 192.168.201.113@tcp [10469.584044] Key type .llcrypt unregistered [10469.585940] Key type ._llcrypt unregistered [10496.962701] Key type ._llcrypt registered [10496.966496] Key type .llcrypt registered [10497.104751] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10498.645806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10498.760294] alg: No test for adler32 (adler32-zlib) [10499.951451] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [10500.314481] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10502.111213] Key type lgssc registered [10503.298969] Lustre: Echo OBD driver; http://www.lustre.org/ [10513.524451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10513.557506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.195079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10520.250378] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10525.681231] LustreError: 329397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.704774] LustreError: 329397:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10530.102928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.566454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10535.723426] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10544.777718] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10552.003658] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10556.247166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.252020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.264546] Lustre: Skipped 1 previous similar message [10556.289647] Lustre: Skipped 1 previous similar message [10561.507357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10561.520425] Lustre: Skipped 1 previous similar message [10563.262246] Lustre: server umount lustre-MDT0000 complete [10566.624312] LustreError: 329394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10566.655123] LustreError: 329394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10568.878434] LustreError: 329378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490195 with bad export cookie 7003576744573480826 [10568.881843] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.895428] LustreError: 329378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10569.565210] Lustre: server umount lustre-MDT0001 complete [10590.640220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10591.108408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10591.196575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10591.733917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.142069] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10611.251707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.392786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10611.496326] Lustre: 331986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10611.883970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.709211] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10628.273402] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10635.467921] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10646.961452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10647.189962] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10647.201486] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10647.213438] LustreError: 333875:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10647.219840] LustreError: 333875:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10647.224847] LustreError: 333875:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10647.291009] Lustre: server umount lustre-OST0000 complete [10647.294970] LustreError: 333875:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10659.046269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.408415] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10659.712963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10667.186157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10669.298435] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10677.932406] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10688.111699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10688.206954] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10688.292467] Lustre: lustre-OST0001: new disk, initializing [10688.303355] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10688.314228] Lustre: 335416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10688.448275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10696.821994] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10697.368960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10697.382282] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10697.502146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10705.986034] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10713.057502] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10713.059314] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10713.070562] LustreError: Skipped 1 previous similar message [10713.071519] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10713.136561] Lustre: Skipped 1 previous similar message [10715.597776] Lustre: server umount lustre-OST0001 complete [10718.187855] LustreError: 334340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.222933] LustreError: 334340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10723.297884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10723.307972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.313027] LustreError: 334343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10723.334182] Lustre: Skipped 1 previous similar message [10723.334485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10723.334489] Lustre: Skipped 1 previous similar message [10723.386817] LustreError: 334343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10727.188862] Lustre: server umount lustre-OST0000 complete [10733.556301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.566630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10733.575464] Lustre: Skipped 2 previous similar messages [10743.777768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10743.786822] Lustre: Skipped 3 previous similar messages [10747.359206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10747.763426] Lustre: server umount lustre-MDT0000 complete [10748.897551] LustreError: 331995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.947655] LustreError: 331995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10752.590667] LustreError: 331981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490378 with bad export cookie 7003576744573481372 [10752.600522] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10753.209298] Lustre: server umount lustre-MDT0001 complete [10766.360659] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10769.970114] Key type lgssc unregistered [10770.329154] LNet: 337376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10770.340789] LNetError: 337376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10771.371983] LNet: Removed LNI 192.168.201.113@tcp [10772.696285] Key type .llcrypt unregistered [10772.697945] Key type ._llcrypt unregistered [10855.746225] Key type ._llcrypt registered [10855.749993] Key type .llcrypt registered [10856.015985] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [10873.590835] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10875.463543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10875.595167] alg: No test for adler32 (adler32-zlib) [10876.772937] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [10877.331216] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10879.015420] Key type lgssc registered [10880.684859] Lustre: Echo OBD driver; http://www.lustre.org/ [10930.285950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10930.307055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.712181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10931.814543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10931.937762] Lustre: lustre-MDT0000: new disk, initializing [10932.042787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10932.065470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10936.699983] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10949.229515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.437375] Lustre: 342312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10949.526574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10949.536755] Lustre: Skipped 1 previous similar message [10949.701429] Lustre: lustre-MDT0001: new disk, initializing [10949.918142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10949.990878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10950.005292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10954.888130] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10966.841693] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10973.760026] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10982.405780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.606335] Lustre: lustre-OST0000: new disk, initializing [10982.608429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10982.610845] Lustre: 344283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10982.691656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10989.727838] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10992.823282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10992.857912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10992.992607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11000.781734] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11007.205371] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11007.629493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11014.238096] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11014.831528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11018.726451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11018.729399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11018.738832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11018.747232] LustreError: Skipped 1 previous similar message [11023.853106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11023.870162] Lustre: Skipped 2 previous similar messages [11024.338561] Lustre: server umount lustre-OST0000 complete [11032.033669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11032.038089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11032.046712] Lustre: Skipped 1 previous similar message [11032.051176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.081555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.085551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.996377] Lustre: server umount lustre-MDT0000 complete [11040.741328] LustreError: 342304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490666 with bad export cookie 643197266969968007 [11040.749464] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11040.754794] LustreError: 342304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11041.374848] Lustre: server umount lustre-MDT0001 complete [11051.295848] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:44:36 (1789490676) [11059.743993] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:44:45 (1789490685) [11069.809866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.402648] LustreError: 347042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.419132] LustreError: 347042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11070.494787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11075.552642] LustreError: 347043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.152640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11080.676594] LustreError: 347042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.798459] LustreError: 347043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.320426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.849099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11091.744969] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11102.936220] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11109.860490] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11121.475486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11122.255575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11131.088996] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11141.341907] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11150.490521] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11152.531100] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11154.764158] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11156.750896] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11159.175943] Lustre: Failing over lustre-MDT0000 [11159.361075] Lustre: server umount lustre-MDT0000 complete [11161.572761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.574890] LustreError: 347771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.594072] Lustre: Skipped 1 previous similar message [11161.628906] LustreError: 347771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11171.817183] LustreError: 347037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.836552] LustreError: 347037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11176.863188] Lustre: 340533:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789490787/real 1789490787] req@000000006cda2a55 x1876416793751808/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789490803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11176.883724] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.014540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11187.185028] LustreError: 340529:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007fa27bc4 x1876416793757184/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11187.636640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11187.727534] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11190.208164] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11192.851923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11192.966345] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11193.113106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11194.977699] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11208.978545] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11211.239919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11215.710857] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11217.770324] Lustre: Failing over lustre-MDT0000 [11218.134697] Lustre: server umount lustre-MDT0000 complete [11218.400887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11218.407296] LustreError: 347043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.409327] Lustre: Skipped 2 previous similar messages [11218.413697] LustreError: 347043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [11234.786407] Lustre: 340533:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789490844/real 1789490844] req@0000000076835738 x1876416793774336/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789490860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11234.811268] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11238.302658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11244.560913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11244.655834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11246.477685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11249.671765] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11249.687500] Lustre: Skipped 2 previous similar messages [11249.744074] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11249.825806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11251.434135] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11263.828380] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11265.798693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11275.233405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11275.235532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.271623] Lustre: Skipped 1 previous similar message [11275.277848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11275.284914] Lustre: Skipped 3 previous similar messages [11280.048552] Lustre: server umount lustre-MDT0000 complete [11280.352301] LustreError: 348132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.361242] LustreError: 348132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [11284.123247] LustreError: 347024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789490910 with bad export cookie 643197266969970723 [11284.130878] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.137637] LustreError: 347024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11284.510365] Lustre: server umount lustre-MDT0001 complete [11298.605591] Lustre: server umount lustre-OST0000 complete [11322.358337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.105912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11327.701782] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11341.891708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.989073] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11357.570639] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11365.014594] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.796503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11379.301136] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11389.955247] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11394.749394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11405.304629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11405.316937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11405.326884] Lustre: Skipped 2 previous similar messages [11405.340975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11405.352921] Lustre: Skipped 1 previous similar message [11409.725191] Lustre: server umount lustre-OST0000 complete [11419.620651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11425.768841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11425.789146] Lustre: Skipped 4 previous similar messages [11430.367216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11430.883079] LustreError: 353382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.898418] LustreError: 353382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11430.927724] Lustre: server umount lustre-MDT0000 complete [11434.765040] LustreError: 353367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491060 with bad export cookie 643197266969971472 [11434.781616] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11434.785956] LustreError: 353367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11435.215866] Lustre: server umount lustre-MDT0001 complete [11446.516985] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11449.487254] Key type lgssc unregistered [11449.696972] LNet: 356946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11449.707588] LNetError: 356946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11449.746594] LNet: Removed LNI 192.168.201.113@tcp [11450.411867] Key type .llcrypt unregistered [11450.414416] Key type ._llcrypt unregistered [11474.737503] Key type ._llcrypt registered [11474.739288] Key type .llcrypt registered [11474.857271] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11475.984369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11476.010183] alg: No test for adler32 (adler32-zlib) [11477.076131] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [11477.288862] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11478.951205] Key type lgssc registered [11480.246813] Lustre: Echo OBD driver; http://www.lustre.org/ [11488.517062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11488.546484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.985262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11494.751696] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11500.512242] LustreError: 358440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11500.539118] LustreError: 358440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11503.444618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.841562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11508.976182] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11517.280194] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11523.451706] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11531.263797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11531.739558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11535.873204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11537.560267] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11545.565339] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11560.417307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11560.425671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.442838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.081647] Lustre: server umount lustre-MDT0000 complete [11563.491116] LustreError: 359167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.496809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.520212] LustreError: 359167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11563.546625] Lustre: Skipped 1 previous similar message [11565.804675] LustreError: 358420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491191 with bad export cookie 4795003500247439054 [11565.825913] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.111353] Lustre: server umount lustre-MDT0001 complete [11580.410423] Lustre: server umount lustre-OST0000 complete [11593.686417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.282333] LustreError: 362171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.310482] LustreError: 362171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11594.385354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11599.096621] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11599.848032] LustreError: 362172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.947878] LustreError: 362171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.487540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.006661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11612.474657] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11619.850809] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11626.678123] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.188313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.830791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11639.882205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11645.015793] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11653.420410] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11669.986795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.991591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11670.016459] Lustre: Skipped 1 previous similar message [11672.837558] Lustre: server umount lustre-MDT0000 complete [11675.112549] LustreError: 362171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.126830] LustreError: 362171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11676.480586] LustreError: 362153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491302 with bad export cookie 4795003500247440167 [11676.484674] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.489531] LustreError: 362153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11676.739915] Lustre: server umount lustre-MDT0001 complete [11690.212736] Lustre: server umount lustre-OST0000 complete [11709.179422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.614966] LustreError: 366082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11709.677617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11715.831116] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11730.500743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.411491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11750.118893] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11757.622517] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11767.064794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11767.403713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.415156] Lustre: Skipped 1 previous similar message [11774.800694] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11779.739453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11782.784957] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.304404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11790.313401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11790.322295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11790.322307] Lustre: Skipped 2 previous similar messages [11790.375221] Lustre: Skipped 2 previous similar messages [11793.674550] Lustre: server umount lustre-OST0000 complete [11800.558684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.565614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11800.600676] Lustre: Skipped 1 previous similar message [11800.622727] Lustre: Skipped 2 previous similar messages [11804.992908] Lustre: server umount lustre-MDT0000 complete [11805.664188] LustreError: 366083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11805.688138] LustreError: 366083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11809.370504] LustreError: 366063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491435 with bad export cookie 4795003500247441189 [11809.372542] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11809.390960] LustreError: 366063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11809.663261] Lustre: server umount lustre-MDT0001 complete [11820.830342] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11824.117982] Key type lgssc unregistered [11824.393913] LNet: 369616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11824.402837] LNetError: 369616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11824.422391] LNet: Removed LNI 192.168.201.113@tcp [11825.295798] Key type .llcrypt unregistered [11825.298381] Key type ._llcrypt unregistered [11857.434183] Key type ._llcrypt registered [11857.447544] Key type .llcrypt registered [11857.856941] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11860.163958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11860.258088] alg: No test for adler32 (adler32-zlib) [11861.777109] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [11862.384472] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11864.160313] Key type lgssc registered [11865.694451] Lustre: Echo OBD driver; http://www.lustre.org/ [11876.218487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11876.225380] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.238229] LustreError: 371055:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11876.275517] LustreError: 371055:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11876.289085] LustreError: 371055:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11876.295255] LustreError: 371055:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11876.301439] LustreError: 371055:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11886.090975] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11886.099569] LustreError: 371453:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11886.138620] LustreError: 371453:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11886.153351] LustreError: 371453:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11886.159407] LustreError: 371453:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11886.166376] LustreError: 371453:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11908.950752] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11912.306637] Key type lgssc unregistered [11912.677261] LNet: 372456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11912.683871] LNetError: 372456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11912.697244] LNet: Removed LNI 192.168.201.113@tcp [11913.496800] Key type .llcrypt unregistered [11913.502691] Key type ._llcrypt unregistered [11934.302968] Key type ._llcrypt registered [11934.304854] Key type .llcrypt registered [11934.410703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [11951.306497] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11953.051706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11953.170424] alg: No test for adler32 (adler32-zlib) [11954.181335] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [11954.360923] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11956.023214] Key type lgssc registered [11957.374233] Lustre: Echo OBD driver; http://www.lustre.org/ [12006.937627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12006.978789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.442878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12008.484975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12008.597503] Lustre: lustre-MDT0000: new disk, initializing [12008.699674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12008.723202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12014.141591] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12025.517661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12025.708235] Lustre: 375619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12025.770203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12025.777160] Lustre: Skipped 1 previous similar message [12025.871142] Lustre: lustre-MDT0001: new disk, initializing [12025.975540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12026.027823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12026.039492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12031.064751] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12044.796306] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12050.885459] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12058.881538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12059.094537] Lustre: lustre-OST0000: new disk, initializing [12059.099372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12059.105895] Lustre: 377600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12059.258807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12065.291948] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12068.456077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12068.485078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12068.620079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12076.585371] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12082.727941] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12082.925878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12088.793238] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12089.256991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12094.436437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12094.444247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12094.448333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12094.455134] LustreError: Skipped 1 previous similar message [12094.488422] Lustre: Skipped 1 previous similar message [12099.292153] Lustre: server umount lustre-OST0000 complete [12104.672964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.677381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12104.688669] Lustre: Skipped 1 previous similar message [12104.696574] Lustre: Skipped 2 previous similar messages [12109.792951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12109.805920] Lustre: Skipped 1 previous similar message [12111.137904] Lustre: server umount lustre-MDT0000 complete [12115.761228] LustreError: 375613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491741 with bad export cookie 985172672431834631 [12115.771879] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12116.569471] Lustre: server umount lustre-MDT0001 complete [12127.294561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12129.557160] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:02:34 (1789491754) [12136.493319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12137.186312] LustreError: 380061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.233356] LustreError: 380061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12137.309844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12142.303732] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12142.578762] LustreError: 380062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12147.695072] LustreError: 380061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12152.811129] LustreError: 380062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12153.656888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12154.517488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12160.441288] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12168.260753] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12173.792748] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12181.749065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.275237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12188.567784] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12197.358839] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12205.404297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12205.627836] Lustre: lustre-OST0001: new disk, initializing [12205.631487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12205.637469] Lustre: 382901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12205.774834] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12212.945169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12212.962332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12213.165587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12213.966780] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12226.441963] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12233.708145] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12233.731209] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.760174] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12233.760575] Lustre: Skipped 1 previous similar message [12233.803364] Lustre: Skipped 1 previous similar message [12238.342305] Lustre: server umount lustre-OST0001 complete [12238.842060] LustreError: 383207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.870273] LustreError: 383207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12249.061422] LustreError: 383207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.081877] LustreError: 383207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12259.303034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.306068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12259.329462] Lustre: Skipped 1 previous similar message [12259.362821] Lustre: Skipped 2 previous similar messages [12269.554594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12269.558759] LustreError: 381887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.567756] Lustre: Skipped 3 previous similar messages [12269.594660] LustreError: 381887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [12273.119141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12273.420770] Lustre: server umount lustre-OST0000 complete [12279.779051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12284.836540] Lustre: server umount lustre-MDT0000 complete [12289.457531] LustreError: 380044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789491915 with bad export cookie 985172672431835520 [12289.463058] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12289.481116] LustreError: 380044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12296.068580] Lustre: server umount lustre-MDT0001 complete [12310.556935] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12314.646682] Key type lgssc unregistered [12315.033475] LNet: 384891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12315.039979] LNetError: 384891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12316.071774] LNet: Removed LNI 192.168.201.113@tcp [12317.384711] Key type .llcrypt unregistered [12317.388169] Key type ._llcrypt unregistered [12386.406847] Key type ._llcrypt registered [12386.414436] Key type .llcrypt registered [12386.644788] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12389.895216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12389.988121] alg: No test for adler32 (adler32-zlib) [12391.805534] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [12392.253936] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12394.153111] Key type lgssc registered [12396.156812] Lustre: Echo OBD driver; http://www.lustre.org/ [12407.033715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12407.049781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12408.337951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12408.350991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12408.454597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12413.990522] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12426.530664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.710334] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12426.786356] Lustre: 387691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12427.243687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12434.319975] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12446.976681] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12454.788190] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12457.967267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12457.977695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12457.994825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12462.057679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.062129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12462.089222] Lustre: Skipped 1 previous similar message [12464.047229] Lustre: server umount lustre-MDT0000 complete [12469.336526] LustreError: 387681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492095 with bad export cookie 11301434330769147842 [12469.347878] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12469.360721] LustreError: 387681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12469.765114] Lustre: server umount lustre-MDT0001 complete [12481.936429] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:08:27 (1789492107) [12490.328436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.900322] LustreError: 390202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.924834] LustreError: 390202:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12490.977393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12496.359449] LustreError: 390203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.519533] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12501.479289] LustreError: 390202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.598436] LustreError: 390203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.739410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.542469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12514.527665] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12524.707106] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12534.831548] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12542.462997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12542.657406] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12542.984053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12548.870286] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12558.608287] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12567.762078] Lustre: Permanently deactivating lustre-OST0000 [12567.778505] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12577.857901] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12579.278188] LustreError: 393192:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12579.292035] LustreError: 393192:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12579.299908] LustreError: 393192:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12583.651652] LustreError: 393240:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12583.660962] LustreError: 393240:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12592.098767] Lustre: server umount lustre-OST0000 complete [12596.193352] LustreError: 393492:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12596.204076] LustreError: 393492:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12600.800532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12600.821165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12600.848196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12602.856734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12602.861193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.973449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12607.985612] Lustre: Skipped 2 previous similar messages [12609.199673] Lustre: server umount lustre-MDT0000 complete [12613.094832] LustreError: 390203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12613.140155] LustreError: 390203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12615.038628] LustreError: 392896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492241 with bad export cookie 11301434330769148423 [12615.039403] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12615.058456] LustreError: 392896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12625.673765] Lustre: server umount lustre-MDT0001 complete [12642.083460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12649.859554] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12652.288562] LustreError: 394653:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12658.786994] Lustre: 394992:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.113@tcp [12661.159426] Lustre: server umount MGS complete [12676.523514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.197633] LustreError: 395716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.229935] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12677.238526] Lustre: Skipped 1 previous similar message [12677.337488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.405504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12694.174548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.643136] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12694.788233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12700.768760] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12710.814553] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12718.247160] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12720.907777] Lustre: Permanently reactivating lustre-OST0000 [12720.954127] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12720.965649] Lustre: Skipped 2 previous similar messages [12747.754250] LustreError: 397451:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12757.957561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12758.441604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12765.025782] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12773.967960] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12783.072922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12783.077526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.094627] LustreError: Skipped 1 previous similar message [12783.095968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12783.168457] Lustre: Skipped 1 previous similar message [12788.927985] Lustre: server umount lustre-OST0000 complete [12797.415622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12797.429644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.542058] Lustre: server umount lustre-MDT0000 complete [12803.555320] LustreError: 395711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12803.572105] LustreError: 395711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12805.477280] LustreError: 395698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492431 with bad export cookie 11301434330769149473 [12805.492160] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12805.498375] LustreError: 395698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12806.488265] Lustre: server umount lustre-MDT0001 complete [12821.825538] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12825.454649] Key type lgssc unregistered [12825.902138] LNet: 399987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12825.912126] LNetError: 399987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12825.942611] LNet: Removed LNI 192.168.201.113@tcp [12826.930781] Key type .llcrypt unregistered [12826.936327] Key type ._llcrypt unregistered [12853.067720] Key type ._llcrypt registered [12853.069610] Key type .llcrypt registered [12853.222380] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [12874.809827] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12876.684691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12876.712643] alg: No test for adler32 (adler32-zlib) [12878.002517] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [12878.463218] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12880.239348] Key type lgssc registered [12881.829900] Lustre: Echo OBD driver; http://www.lustre.org/ [12929.403144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12929.432471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12930.756200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12930.794427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12930.870202] Lustre: lustre-MDT0000: new disk, initializing [12931.017071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12931.073537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12937.306158] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12951.258848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12951.447838] Lustre: 403156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12951.500572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12951.503146] Lustre: Skipped 1 previous similar message [12951.572513] Lustre: lustre-MDT0001: new disk, initializing [12951.678256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12951.709805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12951.719084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12957.787976] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12971.904932] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12978.520554] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12986.586848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12987.143312] Lustre: lustre-OST0000: new disk, initializing [12987.149847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12987.162697] Lustre: 405126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12987.295835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12995.718407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12995.746362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12995.831924] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12995.927926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13009.270877] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13015.104346] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13015.368634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13021.520194] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13021.785994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13026.786420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.794293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13026.809934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13030.787284] Lustre: server umount lustre-OST0000 complete [13037.026034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13037.027349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13037.042094] Lustre: Skipped 2 previous similar messages [13037.064841] Lustre: Skipped 2 previous similar messages [13042.615648] Lustre: server umount lustre-MDT0000 complete [13047.266696] LustreError: 405339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.297961] LustreError: 405339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13047.487589] LustreError: 403146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492673 with bad export cookie 245207156343851908 [13047.504810] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13047.517956] LustreError: 403146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13048.100482] Lustre: server umount lustre-MDT0001 complete [13058.998391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13061.550799] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:18:06 (1789492686) [13070.746525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.196600] LustreError: 407589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.310918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13076.452943] LustreError: 407590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13076.491281] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13081.570905] LustreError: 407589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13086.253084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.603307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13093.815530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13104.638822] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13112.950632] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13122.312066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13122.880248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13130.907218] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13141.190360] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13143.797125] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13146.820455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13156.860776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.874962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13156.889648] Lustre: Skipped 1 previous similar message [13156.890809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13156.930486] Lustre: Skipped 1 previous similar message [13161.952252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13161.967958] Lustre: Skipped 1 previous similar message [13167.073873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13167.084560] Lustre: Skipped 1 previous similar message [13168.095362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13168.449422] Lustre: server umount lustre-OST0000 complete [13173.734861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13173.744501] LustreError: Skipped 1 previous similar message [13173.751368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.313285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13177.315772] Lustre: Skipped 1 previous similar message [13179.865317] Lustre: server umount lustre-MDT0000 complete [13182.431833] LustreError: 408318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13182.449175] LustreError: 408318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13183.779984] LustreError: 407570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789492809 with bad export cookie 245207156343852783 [13183.800022] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13183.802905] LustreError: 407570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13184.151579] Lustre: server umount lustre-MDT0001 complete [13200.186733] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13205.027581] Key type lgssc unregistered [13205.488782] LNet: 411319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13205.499495] LNetError: 411319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13205.514932] LNet: Removed LNI 192.168.201.113@tcp [13206.784322] Key type .llcrypt unregistered [13206.788081] Key type ._llcrypt unregistered [13240.924983] Key type ._llcrypt registered [13240.926191] Key type .llcrypt registered [13241.044291] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13241.392412] Key type .llcrypt unregistered [13241.393895] Key type ._llcrypt unregistered [13265.363868] Key type ._llcrypt registered [13265.365406] Key type .llcrypt registered [13265.551526] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13267.291613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13267.342958] alg: No test for adler32 (adler32-zlib) [13268.492722] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [13268.757158] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13270.479212] Key type lgssc registered [13272.077707] Lustre: Echo OBD driver; http://www.lustre.org/ [13282.664651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13282.708323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13284.506297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13290.982651] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13294.054570] LustreError: 413528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13294.074788] LustreError: 413528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13299.180724] LustreError: 413529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13301.990347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13302.372973] LustreError: 413528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.402045] LustreError: 413528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13302.405983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13307.875321] LustreError: 413525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.909937] LustreError: 413525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13308.948646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13311.307599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13314.027016] LustreError: 413529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13314.056498] LustreError: 413529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13321.645566] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13324.265651] LustreError: 413529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.289267] LustreError: 413529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13328.803073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13329.240492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13336.544653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13349.867152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.870119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13349.884758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13349.885537] Lustre: Skipped 1 previous similar message [13349.926725] Lustre: Skipped 1 previous similar message [13354.986209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13354.996661] Lustre: Skipped 1 previous similar message [13360.098858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13360.114764] Lustre: Skipped 1 previous similar message [13361.121375] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13361.374209] Lustre: server umount lustre-OST0000 complete [13370.339890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13370.340619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13370.361997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13370.364715] LustreError: Skipped 1 previous similar message [13370.381320] Lustre: Skipped 1 previous similar message [13372.794780] Lustre: server umount lustre-MDT0000 complete [13375.456744] LustreError: 414521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13375.468401] LustreError: 414521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13377.730510] LustreError: 413508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493003 with bad export cookie 7378152763017909519 [13377.750150] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13378.089304] Lustre: server umount lustre-MDT0001 complete [13393.523612] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13396.924742] Key type lgssc unregistered [13397.401688] LNet: 416682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13397.418456] LNetError: 416682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13397.445965] LNet: Removed LNI 192.168.201.113@tcp [13398.666624] Key type .llcrypt unregistered [13398.670894] Key type ._llcrypt unregistered [13432.348538] Key type ._llcrypt registered [13432.352704] Key type .llcrypt registered [13432.546904] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13434.238630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13434.343305] alg: No test for adler32 (adler32-zlib) [13435.888637] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [13436.280979] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13438.031331] Key type lgssc registered [13439.796784] Lustre: Echo OBD driver; http://www.lustre.org/ [13450.064049] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13450.085915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13465.695449] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13470.815417] LustreError: 418075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [13497.567220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13505.269550] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13515.656948] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13524.945519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13525.371717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13525.428211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13530.593628] LustreError: 418101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13530.605275] LustreError: 418101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13531.120053] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13533.698245] LustreError: 418100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13533.722655] LustreError: 418100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13538.796746] LustreError: 418101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13538.833499] LustreError: 418101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13542.719873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13543.169686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13548.933139] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13558.890903] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13567.726947] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13576.676090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13576.694466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13576.728282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13579.236654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13579.254439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13579.266273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13582.651814] Lustre: server umount lustre-OST0000 complete [13587.790938] Lustre: server umount lustre-MDT0000 complete [13589.472859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13589.476697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13589.495661] LustreError: 419507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13589.523831] LustreError: 419507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13592.435192] LustreError: 418082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493218 with bad export cookie 3527105358052851589 [13592.460086] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13593.018989] Lustre: server umount lustre-MDT0001 complete [13607.392504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13610.717350] Key type lgssc unregistered [13611.075277] LNet: 421648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13611.086564] LNetError: 421648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13611.107221] LNet: Removed LNI 192.168.201.113@tcp [13612.408990] Key type .llcrypt unregistered [13612.411425] Key type ._llcrypt unregistered [13651.346838] Key type ._llcrypt registered [13651.353668] Key type .llcrypt registered [13651.534895] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13653.480098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13653.579453] alg: No test for adler32 (adler32-zlib) [13654.763432] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [13655.171989] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13657.015334] Key type lgssc registered [13658.338296] Lustre: Echo OBD driver; http://www.lustre.org/ [13667.713416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13667.743656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13669.213308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13674.783467] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13679.586418] LustreError: 423067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13679.620565] LustreError: 423067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13684.704971] LustreError: 423066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13685.929202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13686.473615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13692.031602] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13703.321043] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13711.609330] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13721.561182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13722.045198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13730.473584] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13739.445280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13742.081569] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13748.192730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13748.203142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13748.222180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.734541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.739675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13749.747162] Lustre: Skipped 1 previous similar message [13749.762329] Lustre: Skipped 2 previous similar messages [13750.185771] Lustre: server umount lustre-MDT0000 complete [13750.285612] LustreError: 423061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13753.591712] LustreError: 425646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1789493379 with bad export cookie 15737404964771172360 [13753.618102] LustreError: 425646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13754.857117] LustreError: 424308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13754.896076] LustreError: 424308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13756.602599] LustreError: 423048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789493382 with bad export cookie 15737404964771171464 [13756.608842] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13756.620680] LustreError: 423048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13756.896495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13756.901870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13756.947395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13759.971633] LustreError: 423066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13759.977799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13759.995964] LustreError: 423066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13762.876231] Lustre: server umount lustre-OST0000 complete [13770.800803] Lustre: server umount lustre-MDT0001 complete [13787.160625] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13790.184543] Key type lgssc unregistered [13790.585970] LNet: 426754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13790.595454] LNetError: 426754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13790.619813] LNet: Removed LNI 192.168.201.113@tcp [13792.055471] Key type .llcrypt unregistered [13792.058793] Key type ._llcrypt unregistered [13824.391106] Key type ._llcrypt registered [13824.393639] Key type .llcrypt registered [13824.503611] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13825.233559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13825.269957] alg: No test for adler32 (adler32-zlib) [13826.515215] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [13826.701831] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13828.359221] Key type lgssc registered [13829.710555] Lustre: Echo OBD driver; http://www.lustre.org/ [13839.377420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13839.395301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13840.896436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13845.850620] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13850.084522] LustreError: 428166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13850.092040] LustreError: 428166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13855.214028] LustreError: 428167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.736570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13857.155647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13862.049768] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13872.102303] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13882.644300] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13890.681604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13891.068462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13899.054581] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13908.613194] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13910.587871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13918.690489] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13918.701264] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13918.717384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13920.742244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13920.744772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13924.605129] Lustre: server umount lustre-MDT0001 complete [13925.867162] LustreError: 428162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13925.895574] LustreError: 428162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13930.996613] LustreError: 429269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13931.020620] LustreError: 429269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13936.097456] LustreError: 428167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13936.101832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13936.146350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13936.155221] Lustre: Skipped 2 previous similar messages [13939.727737] Lustre: server umount lustre-OST0000 complete [13944.919158] Lustre: server umount lustre-MDT0000 complete [13965.203727] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13968.804353] Key type lgssc unregistered [13969.065881] LNet: 431854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13969.072719] LNetError: 431854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13969.084218] LNet: Removed LNI 192.168.201.113@tcp [13970.305311] Key type .llcrypt unregistered [13970.306700] Key type ._llcrypt unregistered [14004.034618] Key type ._llcrypt registered [14004.049297] Key type .llcrypt registered [14004.220693] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [14025.630840] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14027.088108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14027.205792] alg: No test for adler32 (adler32-zlib) [14028.430391] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [14028.785791] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14030.463249] Key type lgssc registered [14032.353655] Lustre: Echo OBD driver; http://www.lustre.org/ [14087.924954] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14108.163738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14108.190748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14109.567802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14109.588191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14109.667755] Lustre: lustre-MDT0000: new disk, initializing [14109.762129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14109.787603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14116.317572] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14136.960950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14137.273814] Lustre: lustre-OST0000: new disk, initializing [14137.284585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14137.300640] Lustre: Skipped 1 previous similar message [14137.311725] Lustre: 436911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14137.464716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14143.167478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14143.183782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14143.241235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14146.197830] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14158.424535] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14180.736611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14180.959570] Lustre: 435893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14181.067445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14181.187923] Lustre: lustre-MDT0001: new disk, initializing [14181.380744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14181.441871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14181.445876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14187.976614] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14200.364677] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14200.576857] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14206.776545] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14207.014452] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14213.096976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14213.098433] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14213.117886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14217.180700] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [14217.188064] Lustre: Skipped 1 previous similar message [14218.208745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14218.217380] Lustre: Skipped 1 previous similar message [14219.424662] Lustre: server umount lustre-MDT0001 complete [14228.468113] LustreError: 435900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14228.489187] LustreError: 435900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14233.575055] LustreError: 437623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14233.603421] LustreError: 437623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14238.688383] LustreError: 435904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14238.690417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14238.741346] Lustre: Skipped 1 previous similar message [14238.750757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14243.808163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14243.808284] LustreError: 438813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14245.063318] Lustre: server umount lustre-OST0000 complete [14248.931653] LustreError: 435904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14256.823733] Lustre: server umount lustre-MDT0000 complete [14275.871374] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14279.561276] Key type lgssc unregistered [14280.012615] LNet: 440663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14280.025583] LNetError: 440663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14280.053887] LNet: Removed LNI 192.168.201.113@tcp [14281.175562] Key type .llcrypt unregistered [14281.177867] Key type ._llcrypt unregistered [14337.418620] Key type ._llcrypt registered [14337.421653] Key type .llcrypt registered [14337.597653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14338.177474] Key type .llcrypt unregistered [14338.179752] Key type ._llcrypt unregistered [14361.925263] Key type ._llcrypt registered [14361.927084] Key type .llcrypt registered [14362.076640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14363.220389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14363.289942] alg: No test for adler32 (adler32-zlib) [14364.643394] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [14364.996938] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14366.671248] Key type lgssc registered [14368.151791] Lustre: Echo OBD driver; http://www.lustre.org/ [14403.129421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14403.176746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14404.820080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14404.866073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14405.085598] Lustre: lustre-MDT0000: new disk, initializing [14405.281318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14405.320650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14411.268554] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14426.687741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14426.830744] Lustre: 444041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14426.899926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14426.903837] Lustre: Skipped 1 previous similar message [14427.013766] Lustre: lustre-MDT0001: new disk, initializing [14427.176123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14427.227796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14427.244623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14432.746708] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14445.276726] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14453.450931] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14463.890900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14464.445688] Lustre: lustre-OST0000: new disk, initializing [14464.461821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14464.477219] Lustre: 446010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14464.663845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14472.789283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14472.805616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14473.047462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14474.428971] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14488.949157] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14498.787752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14498.788216] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14498.805092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14498.837857] Lustre: Skipped 1 previous similar message [14503.928438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14503.939050] Lustre: Skipped 4 previous similar messages [14504.683261] Lustre: server umount lustre-MDT0000 complete [14510.287144] LustreError: 444031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789494136 with bad export cookie 16670050699283298219 [14510.293386] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14510.302713] LustreError: 444031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14510.820732] Lustre: server umount lustre-MDT0001 complete [14525.729642] Lustre: server umount lustre-OST0000 complete [14539.654818] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:42:44 (1789494164) [14549.421650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14550.094318] LustreError: 448039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14550.134374] LustreError: 448039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14550.195469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14555.622622] LustreError: 448040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14556.032143] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14560.747465] LustreError: 448039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14565.685579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14565.857053] LustreError: 448040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14566.217236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14571.537698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14581.120132] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14589.018536] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14611.250805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14611.430180] LustreError: 449868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14611.542852] Lustre: Found index 0 for lustre-OST0000, updating log [14611.655655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14619.922962] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14620.228034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14631.792057] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14640.616433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14640.622197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14640.622248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14640.631880] Lustre: Skipped 2 previous similar messages [14645.157829] Lustre: server umount lustre-OST0000 complete [14663.136582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14663.146382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14663.150283] Lustre: Skipped 1 previous similar message [14663.165982] Lustre: Skipped 4 previous similar messages [14667.237976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14667.249338] Lustre: Skipped 1 previous similar message [14669.029431] Lustre: server umount lustre-MDT0000 complete [14673.379215] LustreError: 448034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.401066] LustreError: 448034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14674.460584] LustreError: 448020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789494300 with bad export cookie 16670050699283299605 [14674.467175] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14674.488029] LustreError: 448020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14674.965716] Lustre: server umount lustre-MDT0001 complete [14678.432909] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [14686.811488] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:45:11 (1789494311) [15323.160603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15323.670267] LustreError: 454457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15323.745690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15328.767208] LustreError: 454458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15329.891528] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15333.862425] LustreError: 454457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15340.784951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15341.511107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15347.494819] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15356.865549] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15363.902628] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15383.007357] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [15384.750572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15385.062168] Lustre: Found index 0 for lustre-OST0000, updating log [15385.558119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15393.883116] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15402.960444] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15406.698950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15406.708829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15406.714677] Lustre: Skipped 2 previous similar messages [15412.175642] Lustre: server umount lustre-OST0000 complete [15419.553035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15419.568140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15419.569089] Lustre: Skipped 1 previous similar message [15419.587898] Lustre: Skipped 1 previous similar message [15421.607034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15421.618550] Lustre: Skipped 3 previous similar messages [15424.956575] Lustre: server umount lustre-MDT0000 complete [15425.720499] LustreError: 454453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15425.769454] LustreError: 454453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [15430.385547] LustreError: 454438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495056 with bad export cookie 16670050699283300690 [15430.401284] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15430.406842] LustreError: 454438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15431.060965] Lustre: server umount lustre-MDT0001 complete [15443.425530] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [15465.232274] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15478.064455] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15523.514349] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:59:08 (1789495148) [15545.982977] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15548.939934] Key type lgssc unregistered [15549.378751] LNet: 461077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15549.393486] LNetError: 461077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15648.741907] LNet: Removed LNI 192.168.201.113@tcp [15649.847692] Key type .llcrypt unregistered [15649.851357] Key type ._llcrypt unregistered [15669.067965] Key type ._llcrypt registered [15669.069739] Key type .llcrypt registered [15669.157476] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15670.893601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15670.952895] alg: No test for adler32 (adler32-zlib) [15672.137624] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [15672.385740] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15674.159291] Key type lgssc registered [15675.697813] Lustre: Echo OBD driver; http://www.lustre.org/ [15680.788438] LNet: Added LNI 192.168.201.113@tcp100 [8/256/0/180] [15681.873709] LNet: Added LNI 192.168.201.113@tcp101 [8/256/0/180] [15682.682610] LNet: Added LNI 192.168.201.113@tcp102 [8/256/0/180] [15683.800076] LNet: Added LNI 192.168.201.113@tcp103 [8/256/0/180] [15684.962777] LNet: Added LNI 192.168.201.113@tcp104 [8/256/0/180] [15685.995987] LNet: Added LNI 192.168.201.113@tcp105 [8/256/0/180] [15687.138921] LNet: Added LNI 192.168.201.113@tcp106 [8/256/0/180] [15688.487101] LNet: Added LNI 192.168.201.113@tcp107 [8/256/0/180] [15689.686380] LNet: Added LNI 192.168.201.113@tcp108 [8/256/0/180] [15690.828940] LNet: Added LNI 192.168.201.113@tcp109 [8/256/0/180] [15692.173673] LNet: Added LNI 192.168.201.113@tcp110 [8/256/0/180] [15693.312874] LNet: Added LNI 192.168.201.113@tcp111 [8/256/0/180] [15694.417887] LNet: Added LNI 192.168.201.113@tcp112 [8/256/0/180] [15695.422039] LNet: Added LNI 192.168.201.113@tcp113 [8/256/0/180] [15696.582857] LNet: Added LNI 192.168.201.113@tcp114 [8/256/0/180] [15697.646531] LNet: Added LNI 192.168.201.113@tcp115 [8/256/0/180] [15698.530980] LNet: Added LNI 192.168.201.113@tcp116 [8/256/0/180] [15699.740593] LNet: Added LNI 192.168.201.113@tcp117 [8/256/0/180] [15700.664098] LNet: Added LNI 192.168.201.113@tcp118 [8/256/0/180] [15701.854588] LNet: Added LNI 192.168.201.113@tcp119 [8/256/0/180] [15703.051506] LNet: Added LNI 192.168.201.113@tcp120 [8/256/0/180] [15704.237444] LNet: Added LNI 192.168.201.113@tcp121 [8/256/0/180] [15705.305736] LNet: Added LNI 192.168.201.113@tcp122 [8/256/0/180] [15706.333528] LNet: Added LNI 192.168.201.113@tcp123 [8/256/0/180] [15707.446464] LNet: Added LNI 192.168.201.113@tcp124 [8/256/0/180] [15708.642564] LNet: Added LNI 192.168.201.113@tcp125 [8/256/0/180] [15710.316907] LNet: Added LNI 192.168.201.113@tcp126 [8/256/0/180] [15711.515843] LNet: Added LNI 192.168.201.113@tcp127 [8/256/0/180] [15712.792414] LNet: Added LNI 192.168.201.113@tcp128 [8/256/0/180] [15713.939844] LNet: Added LNI 192.168.201.113@tcp129 [8/256/0/180] [15715.060217] LNet: Added LNI 192.168.201.113@tcp130 [8/256/0/180] [15716.510767] LNet: Added LNI 192.168.201.113@tcp131 [8/256/0/180] [15717.656816] LNet: Added LNI 192.168.201.113@tcp132 [8/256/0/180] [15719.049852] LNet: Added LNI 192.168.201.113@tcp133 [8/256/0/180] [15720.342822] LNet: Added LNI 192.168.201.113@tcp134 [8/256/0/180] [15721.777232] LNet: Added LNI 192.168.201.113@tcp135 [8/256/0/180] [15732.130490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15732.178741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15733.342120] Lustre: 464120:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15733.453571] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15733.469518] LNetError: Protocol error connecting to 192.168.201.113@tcp116 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15733.483287] LNetError: Skipped 6 previous similar messages [15733.575982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15733.615607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15733.747343] Lustre: lustre-MDT0000: new disk, initializing [15733.855140] LNet: 9 local NIs in recovery (showing 9): 192.168.201.113@tcp121, 192.168.201.113@tcp122, 192.168.201.113@tcp124, 192.168.201.113@tcp126, 192.168.201.113@tcp127, 192.168.201.113@tcp128, 192.168.201.113@tcp129, 192.168.201.113@tcp116, 192.168.201.113@tcp125 [15733.991845] Lustre: 464175:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15734.011235] Lustre: 464175:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15734.080727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15734.133544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15734.507984] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15734.533480] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15734.553508] LNetError: Protocol error connecting to 192.168.201.113@tcp112 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15734.571953] LNetError: Skipped 1 previous similar message [15736.996851] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15737.021416] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15737.033802] LNetError: Protocol error connecting to 192.168.201.113@tcp121 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15737.054810] LNetError: Skipped 13 previous similar messages [15739.731640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15748.327847] LNetError: 464152:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15748.342582] LNetError: 464152:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15748.352624] LNetError: Protocol error connecting to 192.168.201.113@tcp128 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15748.378148] LNetError: Skipped 2 previous similar messages [15754.119166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15754.411406] Lustre: 464137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15754.536315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15754.555388] Lustre: Skipped 1 previous similar message [15754.695715] Lustre: lustre-MDT0001: new disk, initializing [15754.742086] Lustre: 464938:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15754.754619] Lustre: 464938:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15754.939108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15754.981840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15754.995572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15760.929491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15762.961988] Lustre: 465282:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15762.973357] Lustre: 465282:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15764.776905] LNetError: 464144:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15764.783887] LNetError: 464144:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15764.803032] LNetError: Protocol error connecting to 192.168.201.113@tcp128 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15772.204727] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15779.938498] LNetError: 464140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15779.956593] LNetError: 464140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15779.962780] LNetError: Protocol error connecting to 192.168.201.113@tcp125 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15779.985282] LNetError: Skipped 4 previous similar messages [15780.350500] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15789.117431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15789.504156] Lustre: lustre-OST0000: new disk, initializing [15789.510507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15789.518914] Lustre: 466111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15789.640410] Lustre: 466116:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15789.648570] Lustre: 466116:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15789.718378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15794.468928] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp128 [15796.320466] LNetError: 461578:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15796.334616] LNetError: 461578:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15796.348910] LNetError: Protocol error connecting to 192.168.201.113@tcp126 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15796.350829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15796.366064] LNetError: Skipped 7 previous similar messages [15797.373791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15797.394767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15797.599944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15808.253463] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15818.210119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15818.218231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15818.263591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15818.275644] Lustre: Skipped 1 previous similar message [15823.335503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15823.350437] Lustre: Skipped 1 previous similar message [15823.868259] Lustre: server umount lustre-OST0000 complete [15828.452537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15828.455249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15828.474398] Lustre: Skipped 2 previous similar messages [15828.494705] Lustre: Skipped 1 previous similar message [15831.136717] LNetError: 464140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15831.152068] LNetError: 464140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15831.178656] LNetError: Protocol error connecting to 192.168.201.113@tcp120 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15831.197353] LNetError: Skipped 9 previous similar messages [15833.575973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15833.589319] Lustre: Skipped 1 previous similar message [15834.948681] Lustre: server umount lustre-MDT0000 complete [15839.257752] LustreError: 464129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789495465 with bad export cookie 13457860200465392999 [15839.267576] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15839.270458] LustreError: 464129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15839.690694] Lustre: server umount lustre-MDT0001 complete [15851.900295] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15855.069536] Key type lgssc unregistered [15855.467394] LNet: 468026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15855.472984] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15855.477735] LNet: Removed LNI 192.168.201.113@tcp [15856.546272] LNet: Removed LNI 192.168.201.113@tcp100 [15856.561878] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15856.570739] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15856.580865] LNet: Removed LNI 192.168.201.113@tcp101 [15857.631273] LNet: Removed LNI 192.168.201.113@tcp102 [15857.641368] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15857.666246] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15857.682453] LNet: Removed LNI 192.168.201.113@tcp103 [15858.719587] LNet: Removed LNI 192.168.201.113@tcp104 [15859.743432] LNet: Removed LNI 192.168.201.113@tcp105 [15859.750727] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15859.762962] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15859.781075] LNet: Removed LNI 192.168.201.113@tcp106 [15860.831170] LNet: Removed LNI 192.168.201.113@tcp107 [15861.855263] LNet: Removed LNI 192.168.201.113@tcp108 [15862.879384] LNet: Removed LNI 192.168.201.113@tcp109 [15863.903195] LNet: Removed LNI 192.168.201.113@tcp110 [15863.905425] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15863.934601] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15863.959421] LNet: Removed LNI 192.168.201.113@tcp111 [15864.991220] LNet: Removed LNI 192.168.201.113@tcp112 [15866.015273] LNet: Removed LNI 192.168.201.113@tcp113 [15867.039682] LNet: Removed LNI 192.168.201.113@tcp114 [15868.063553] LNet: Removed LNI 192.168.201.113@tcp115 [15869.089805] LNet: Removed LNI 192.168.201.113@tcp116 [15870.111149] LNet: Removed LNI 192.168.201.113@tcp117 [15871.135166] LNet: Removed LNI 192.168.201.113@tcp118 [15872.159403] LNet: Removed LNI 192.168.201.113@tcp119 [15872.165299] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15872.173022] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15872.176718] LNet: Removed LNI 192.168.201.113@tcp120 [15873.183210] LNet: Removed LNI 192.168.201.113@tcp121 [15874.207287] LNet: Removed LNI 192.168.201.113@tcp122 [15875.231282] LNet: Removed LNI 192.168.201.113@tcp123 [15876.255352] LNet: Removed LNI 192.168.201.113@tcp124 [15877.281809] LNet: Removed LNI 192.168.201.113@tcp125 [15878.303796] LNet: Removed LNI 192.168.201.113@tcp126 [15879.327582] LNet: Removed LNI 192.168.201.113@tcp127 [15879.330769] LNet: Removed LNI 192.168.201.113@tcp128 [15880.351769] LNet: Removed LNI 192.168.201.113@tcp129 [15880.355762] LNet: Removed LNI 192.168.201.113@tcp130 [15880.361983] LNet: Removed LNI 192.168.201.113@tcp131 [15880.364737] LNet: Removed LNI 192.168.201.113@tcp132 [15880.371425] LNet: Removed LNI 192.168.201.113@tcp133 [15880.373789] LNet: Removed LNI 192.168.201.113@tcp134 [15880.386759] LNet: Removed LNI 192.168.201.113@tcp135 [15881.231858] Key type .llcrypt unregistered [15881.238689] Key type ._llcrypt unregistered [15908.324223] Key type ._llcrypt registered [15908.327743] Key type .llcrypt registered [15908.520723] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15927.196496] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15928.724959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15928.850436] alg: No test for adler32 (adler32-zlib) [15929.857410] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [15930.129620] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15931.842159] Key type lgssc registered [15933.179755] Lustre: Echo OBD driver; http://www.lustre.org/ [15976.733557] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:06:41 (1789495601) [16000.087377] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16005.055184] Key type lgssc unregistered [16005.531978] LNet: 472113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16005.539875] LNetError: 472113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16005.560321] LNet: Removed LNI 192.168.201.113@tcp [16007.064888] Key type .llcrypt unregistered [16007.068301] Key type ._llcrypt unregistered [16029.752261] Key type ._llcrypt registered [16029.753915] Key type .llcrypt registered [16029.847509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16030.576951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16030.653754] alg: No test for adler32 (adler32-zlib) [16032.246612] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [16032.672897] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16034.551247] Key type lgssc registered [16036.291580] Lustre: Echo OBD driver; http://www.lustre.org/ [16051.794531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16051.901556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16053.217124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16053.264461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16053.385900] Lustre: lustre-MDT0000: new disk, initializing [16053.588975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16053.665563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16059.565787] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16072.221419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16072.435454] Lustre: 473321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16072.579360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16072.583263] Lustre: Skipped 1 previous similar message [16072.683319] Lustre: lustre-MDT0001: new disk, initializing [16072.812861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16072.895057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16072.913951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16079.317031] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16093.154648] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16101.211677] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16109.801790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16110.206865] Lustre: lustre-OST0000: new disk, initializing [16110.213358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16110.225335] Lustre: 475294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16110.429533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16116.873411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16116.881634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16117.053455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16119.154545] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16130.699669] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16135.687247] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lnet_if_list [16143.163449] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16143.211185] LNet: Added LNI 192.168.201.113@tcp10 [8/256/0/180] [16151.761770] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16151.782442] LNet: Added LNI 192.168.201.113@tcp11 [8/256/0/180] [16158.952140] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16158.984609] LNet: Added LNI 192.168.201.113@tcp12 [8/256/0/180] [16164.120949] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16164.151656] LNet: Added LNI 192.168.201.113@tcp13 [8/256/0/180] [16169.609766] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16169.655584] LNet: Added LNI 192.168.201.113@tcp14 [8/256/0/180] [16176.716495] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16176.741687] LNet: Added LNI 192.168.201.113@tcp15 [8/256/0/180] [16183.273943] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16183.323781] LNet: Added LNI 192.168.201.113@tcp16 [8/256/0/180] [16190.947051] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16191.014762] LNet: Added LNI 192.168.201.113@tcp17 [8/256/0/180] [16197.747520] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16197.788408] LNet: Added LNI 192.168.201.113@tcp18 [8/256/0/180] [16203.643748] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16203.659976] LNet: Added LNI 192.168.201.113@tcp19 [8/256/0/180] [16209.696629] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16209.734978] LNet: Added LNI 192.168.201.113@tcp20 [8/256/0/180] [16216.412567] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16216.454383] LNet: Added LNI 192.168.201.113@tcp21 [8/256/0/180] [16222.945923] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16222.989385] LNet: Added LNI 192.168.201.113@tcp22 [8/256/0/180] [16234.236672] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16234.282373] LNet: Added LNI 192.168.201.113@tcp23 [8/256/0/180] [16241.340704] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16241.392885] LNet: Added LNI 192.168.201.113@tcp24 [8/256/0/180] [16248.042424] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16248.083641] LNet: Added LNI 192.168.201.113@tcp25 [8/256/0/180] [16256.498683] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16256.530905] LNet: Added LNI 192.168.201.113@tcp26 [8/256/0/180] [16262.606959] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16262.641765] LNet: Added LNI 192.168.201.113@tcp27 [8/256/0/180] [16269.098479] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16269.154914] LNet: Added LNI 192.168.201.113@tcp28 [8/256/0/180] [16278.080803] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16278.110151] LNet: Added LNI 192.168.201.113@tcp29 [8/256/0/180] [16288.323483] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16288.351473] LNet: Added LNI 192.168.201.113@tcp30 [8/256/0/180] [16298.432246] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16298.485299] LNet: Added LNI 192.168.201.113@tcp31 [8/256/0/180] [16307.794958] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16307.826651] LNet: Added LNI 192.168.201.113@tcp32 [8/256/0/180] [16317.198180] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16317.247736] LNet: Added LNI 192.168.201.113@tcp33 [8/256/0/180] [16326.029590] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16326.064236] LNet: Added LNI 192.168.201.113@tcp34 [8/256/0/180] [16336.708646] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16336.761862] LNet: Added LNI 192.168.201.113@tcp35 [8/256/0/180] [16344.936599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16344.983805] LNet: Added LNI 192.168.201.113@tcp36 [8/256/0/180] [16354.096434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16354.151379] LNet: Added LNI 192.168.201.113@tcp37 [8/256/0/180] [16363.162138] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16363.203478] LNet: Added LNI 192.168.201.113@tcp38 [8/256/0/180] [16373.010246] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16373.065785] LNet: Added LNI 192.168.201.113@tcp39 [8/256/0/180] [16382.916578] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16383.004086] LNet: Added LNI 192.168.201.113@tcp40 [8/256/0/180] [16392.187982] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16392.201512] LNet: Added LNI 192.168.201.113@tcp41 [8/256/0/180] [16400.173158] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16400.238659] LNet: Added LNI 192.168.201.113@tcp42 [8/256/0/180] [16410.521481] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16410.576883] LNet: Added LNI 192.168.201.113@tcp43 [8/256/0/180] [16425.096090] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16425.168025] LNet: Added LNI 192.168.201.113@tcp44 [8/256/0/180] [16435.378970] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16435.428131] LNet: Added LNI 192.168.201.113@tcp45 [8/256/0/180] [16445.001935] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16445.043682] LNet: Added LNI 192.168.201.113@tcp46 [8/256/0/180] [16454.542914] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16454.602327] LNet: Added LNI 192.168.201.113@tcp47 [8/256/0/180] [16464.701906] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16464.741027] LNet: Added LNI 192.168.201.113@tcp48 [8/256/0/180] [16474.155993] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16474.211155] LNet: Added LNI 192.168.201.113@tcp49 [8/256/0/180] [16483.445631] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16483.460882] LNet: Added LNI 192.168.201.113@tcp50 [8/256/0/180] [16511.457051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16511.472246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16511.494319] Lustre: Skipped 2 previous similar messages [16511.539639] Lustre: Skipped 2 previous similar messages [16515.467543] Lustre: server umount lustre-MDT0000 complete [16521.211325] LustreError: 473316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496147 with bad export cookie 6322649250547332689 [16521.221789] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16521.233936] LustreError: 473316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16521.705542] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16521.723282] LustreError: 473328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16521.770494] LustreError: 473328:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [16521.953065] Lustre: server umount lustre-MDT0001 complete [16542.532410] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16555.731909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16566.751690] LustreError: 472593:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008fc495ab x1876422199287936/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16567.234984] LustreError: 475285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16567.338207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16572.400441] LustreError: 475286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16573.456116] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16576.481044] LustreError: 475285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16576.488563] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16581.601966] LustreError: 475286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16583.887662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16584.454064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16586.491268] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16591.298560] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16600.896630] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16608.133988] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16630.753623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16630.761733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16630.780707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16632.805723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16632.820202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16637.083275] Lustre: server umount lustre-OST0000 complete [16642.980653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16642.984276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16643.011468] Lustre: Skipped 1 previous similar message [16643.037626] Lustre: Skipped 2 previous similar messages [16648.162386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16648.186850] Lustre: Skipped 1 previous similar message [16657.375903] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16657.669568] Lustre: server umount lustre-MDT0000 complete [16658.406993] LustreError: 499577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16658.441379] LustreError: 499577:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16662.718948] LustreError: 473314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496288 with bad export cookie 6322649250547335398 [16662.722856] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16662.742291] LustreError: 473314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [16662.968821] Lustre: server umount lustre-MDT0001 complete [16677.585593] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16681.400513] Key type lgssc unregistered [16681.975056] LNet: 502378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16681.989640] LNetError: 502378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16683.039213] LNet: Removed LNI 192.168.201.113@tcp [16683.043253] LNetError: 502378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16683.051993] LNet: Removed LNI 192.168.201.113@tcp10 [16683.066286] LNet: Removed LNI 192.168.201.113@tcp11 [16683.069217] LNet: Removed LNI 192.168.201.113@tcp12 [16683.083256] LNet: Removed LNI 192.168.201.113@tcp13 [16683.084996] LNet: Removed LNI 192.168.201.113@tcp14 [16683.091173] LNet: Removed LNI 192.168.201.113@tcp15 [16683.094436] LNet: Removed LNI 192.168.201.113@tcp16 [16683.102396] LNet: Removed LNI 192.168.201.113@tcp17 [16683.107546] LNet: Removed LNI 192.168.201.113@tcp18 [16683.109268] LNet: Removed LNI 192.168.201.113@tcp19 [16683.113019] LNet: Removed LNI 192.168.201.113@tcp20 [16683.116397] LNet: Removed LNI 192.168.201.113@tcp21 [16683.120598] LNet: Removed LNI 192.168.201.113@tcp22 [16683.126047] LNet: Removed LNI 192.168.201.113@tcp23 [16683.129311] LNet: Removed LNI 192.168.201.113@tcp24 [16683.132143] LNet: Removed LNI 192.168.201.113@tcp25 [16683.136095] LNet: Removed LNI 192.168.201.113@tcp26 [16683.145474] LNet: Removed LNI 192.168.201.113@tcp27 [16683.152933] LNet: Removed LNI 192.168.201.113@tcp28 [16683.157124] LNet: Removed LNI 192.168.201.113@tcp29 [16683.161174] LNet: Removed LNI 192.168.201.113@tcp30 [16683.168393] LNet: Removed LNI 192.168.201.113@tcp31 [16683.174030] LNet: Removed LNI 192.168.201.113@tcp32 [16683.182124] LNet: Removed LNI 192.168.201.113@tcp33 [16683.183656] LNet: Removed LNI 192.168.201.113@tcp34 [16683.198434] LNet: Removed LNI 192.168.201.113@tcp35 [16683.200271] LNet: Removed LNI 192.168.201.113@tcp36 [16683.217871] LNet: Removed LNI 192.168.201.113@tcp37 [16683.224106] LNet: Removed LNI 192.168.201.113@tcp38 [16683.234463] LNet: Removed LNI 192.168.201.113@tcp39 [16683.238403] LNet: Removed LNI 192.168.201.113@tcp40 [16683.243322] LNet: Removed LNI 192.168.201.113@tcp41 [16683.246920] LNet: Removed LNI 192.168.201.113@tcp42 [16683.251561] LNet: Removed LNI 192.168.201.113@tcp43 [16683.259438] LNet: Removed LNI 192.168.201.113@tcp44 [16683.264228] LNet: Removed LNI 192.168.201.113@tcp45 [16683.267561] LNet: Removed LNI 192.168.201.113@tcp46 [16683.272349] LNet: Removed LNI 192.168.201.113@tcp47 [16683.278170] LNet: Removed LNI 192.168.201.113@tcp48 [16683.283199] LNet: Removed LNI 192.168.201.113@tcp49 [16683.297432] LNet: Removed LNI 192.168.201.113@tcp50 [16684.427353] Key type .llcrypt unregistered [16684.432622] Key type ._llcrypt unregistered [16708.243452] Key type ._llcrypt registered [16708.249610] Key type .llcrypt registered [16708.371676] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16726.854545] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16728.235970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16728.320639] alg: No test for adler32 (adler32-zlib) [16729.424293] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [16729.637743] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16731.343810] Key type lgssc registered [16733.057360] Lustre: Echo OBD driver; http://www.lustre.org/ [16776.476583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16776.517438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16777.994306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16778.046536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16778.267156] Lustre: lustre-MDT0000: new disk, initializing [16778.456301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16778.480271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16784.453932] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16796.909836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16797.066677] Lustre: 505573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16797.145315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16797.148546] Lustre: Skipped 1 previous similar message [16797.252476] Lustre: lustre-MDT0001: new disk, initializing [16797.434361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16797.489283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16797.511875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16803.291718] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16814.678883] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16823.967152] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16835.545069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16835.797344] Lustre: *** cfs_fail_loc=90f, val=0*** [16835.961219] Lustre: lustre-OST0000: new disk, initializing [16835.974317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16835.987615] Lustre: 507591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16836.106063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16843.878812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16843.898893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16844.079289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16844.219462] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16856.641051] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16869.864448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16869.865785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16869.874244] LustreError: Skipped 1 previous similar message [16869.876592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16869.883526] Lustre: Skipped 1 previous similar message [16869.894787] Lustre: Skipped 1 previous similar message [16872.330370] Lustre: server umount lustre-OST0000 complete [16879.586698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16879.591970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16879.604230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16886.069882] Lustre: server umount lustre-MDT0000 complete [16890.352212] LustreError: 508263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16890.381758] LustreError: 508263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16891.633745] LustreError: 505564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496517 with bad export cookie 9430568127430009902 [16891.635898] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16891.660366] LustreError: 505564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16892.160139] Lustre: server umount lustre-MDT0001 complete [16905.105988] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16908.077518] Key type lgssc unregistered [16908.343821] LNet: 509502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16908.351410] LNetError: 509502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16908.376497] LNet: Removed LNI 192.168.201.113@tcp [16909.390775] Key type .llcrypt unregistered [16909.392859] Key type ._llcrypt unregistered [16934.564915] Key type ._llcrypt registered [16934.569481] Key type .llcrypt registered [16934.679852] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16935.637550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16935.775831] alg: No test for adler32 (adler32-zlib) [16936.955158] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [16937.379181] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16939.153151] Key type lgssc registered [16940.294512] Lustre: Echo OBD driver; http://www.lustre.org/ [16952.283528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16952.304246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16953.963328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16959.227893] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16964.594041] LustreError: 510771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16964.617127] LustreError: 510771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16968.379849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16969.015317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16974.698438] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16983.578448] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16989.442635] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16996.913790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16997.430318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17006.580048] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17017.493452] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17029.210694] loop0: detected capacity change from 0 to 204800000 [17029.474080] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17029.488411] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17029.502975] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17037.509900] loop0: detected capacity change from 0 to 204800000 [17037.786612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17037.797793] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17037.817514] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17043.414430] loop0: detected capacity change from 0 to 204800000 [17049.725501] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17049.863919] Lustre: fs15246-OST0000: new disk, initializing [17049.867543] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [17049.878338] Lustre: 514365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17049.940726] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17056.722815] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17068.104578] loop1: detected capacity change from 0 to 204800000 [17076.035246] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17076.190979] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [17076.254689] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [17076.317245] Lustre: fs15246-MDT0000: new disk, initializing [17076.474410] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17076.495850] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17076.645931] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [17077.744734] Lustre: 515489:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000006886949 x1876423148094464/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789496719 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [17082.574482] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17086.954642] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17086.962851] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17087.006733] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17109.474817] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17109.499939] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17114.298115] Lustre: server umount fs15246-MDT0000 complete [17114.593098] LustreError: 515819:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17119.716288] LustreError: 510767:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17124.882936] LustreError: 510767:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17128.857638] Lustre: server umount fs15246-OST0000 complete [17145.315273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17145.322954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17145.345812] Lustre: Skipped 1 previous similar message [17145.365930] Lustre: Skipped 1 previous similar message [17149.588824] Lustre: server umount lustre-OST0000 complete [17155.557514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17155.575063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17155.581852] Lustre: Skipped 1 previous similar message [17160.673839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17165.793992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17165.797053] Lustre: Skipped 2 previous similar messages [17168.354283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17168.631737] Lustre: server umount lustre-MDT0000 complete [17170.919449] LustreError: 510771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17170.939231] LustreError: 510771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [17173.604566] LustreError: 515482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789496799 with bad export cookie 16099671849184964659 [17173.608812] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17173.612962] LustreError: 515482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17174.398613] Lustre: server umount lustre-MDT0001 complete [17187.314498] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [17190.641261] Key type lgssc unregistered [17191.157712] LNet: 518591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17191.164863] LNetError: 518591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17191.178702] LNet: Removed LNI 192.168.201.113@tcp [17192.351628] Key type .llcrypt unregistered [17192.355850] Key type ._llcrypt unregistered [17214.296660] Key type ._llcrypt registered [17214.299736] Key type .llcrypt registered [17214.432283] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17216.331299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17216.433709] alg: No test for adler32 (adler32-zlib) [17217.749919] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [17218.167755] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [17220.058424] Key type lgssc registered [17221.104526] Lustre: Echo OBD driver; http://www.lustre.org/ [17276.575909] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [17287.645920] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17330.346734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17330.385422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17331.574595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17331.605200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17331.661612] Lustre: lustre-MDT0000: new disk, initializing [17331.769420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17331.798793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17335.654377] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17349.267174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17349.422313] Lustre: 523070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17349.532394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17349.537449] Lustre: Skipped 1 previous similar message [17349.689818] Lustre: lustre-MDT0001: new disk, initializing [17349.912121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17349.940943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17349.954273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17355.338650] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17368.929291] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17377.441877] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17386.332524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17386.846390] Lustre: lustre-OST0000: new disk, initializing [17386.854274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17386.864888] Lustre: 525039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17387.049688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17394.776761] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17394.788441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17394.789173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17395.034185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17407.176290] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17413.860140] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17414.172680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17422.162291] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17422.568053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17425.890228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17425.899959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17425.924133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17425.932658] Lustre: Skipped 1 previous similar message [17431.010317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17431.028666] Lustre: Skipped 1 previous similar message [17432.064255] Lustre: server umount lustre-OST0000 complete [17441.252821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17441.256961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17441.267191] Lustre: Skipped 2 previous similar messages [17445.844724] Lustre: server umount lustre-MDT0000 complete [17450.607400] LustreError: 523063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789497076 with bad export cookie 12528933001075960855 [17450.611854] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17450.618020] LustreError: 523063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17451.052243] Lustre: server umount lustre-MDT0001 complete [17483.759942] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [17497.161434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17539.491947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17278 sec ========= 14:32:44 (1789497164) [17541.719032] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:32:46 (1789497166) === [17546.024359] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:32:50 (1789497170) === [17599.711551] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [17603.713812] Key type lgssc unregistered [17604.150172] LNet: 531693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17604.166908] LNetError: 531693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17604.194439] LNet: Removed LNI 192.168.201.113@tcp [17605.360525] Key type .llcrypt unregistered [17605.365179] Key type ._llcrypt unregistered