[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516109162 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007532] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012058] SELinux: Initializing. [ 0.013094] *** VALIDATE selinux *** [ 0.021537] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025821] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] *** VALIDATE tmpfs *** [ 0.029454] *** VALIDATE proc *** [ 0.031217] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034182] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038020] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042474] debug: unmapping init [mem 0xffffffff89a59000-0xffffffff89a60fff] [ 0.044155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045532] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.046879] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.047887] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049935] ... event mask: 000000070000000f [ 0.050296] rcu: Hierarchical SRCU implementation. [ 0.052221] smp: Bringing up secondary CPUs ... [ 0.053482] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.058285] smp: Brought up 1 node, 4 CPUs [ 0.059876] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139902] node 0 deferred pages initialised in 76ms [ 0.143094] devtmpfs: initialized [ 0.144144] x86/mm: Memory block size: 128MB [ 0.146431] gcov: version magic: 0x41383552 [ 0.149170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151280] pinctrl core: initialized pinctrl subsystem [ 0.152130] [ 0.152439] ************************************************************* [ 0.153010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154012] ** ** [ 0.155008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.157009] ** This means that this kernel is built to expose internal ** [ 0.158010] ** IOMMU data structures, which may compromise security on ** [ 0.159008] ** your system. ** [ 0.160008] ** ** [ 0.161010] ** If you see this message and you are not debugging the ** [ 0.162010] ** kernel, report this immediately to your vendor! ** [ 0.163009] ** ** [ 0.164007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165007] ************************************************************* [ 0.166503] NET: Registered protocol family 16 [ 0.167349] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174057] cpuidle: using governor menu [ 0.175751] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177479] PCI: Using configuration type 1 for base access [ 0.179149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186024] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189107] cryptd: max_cpu_qlen set to 1000 [ 0.192175] ACPI: Added _OSI(Module Device) [ 0.193009] ACPI: Added _OSI(Processor Device) [ 0.194069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200688] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206372] ACPI: Interpreter enabled [ 0.208068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211017] ACPI: Using IOAPIC for interrupt routing [ 0.213151] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226699] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241349] acpiphp: Slot [2] registered [ 0.243121] acpiphp: Slot [5] registered [ 0.244117] acpiphp: Slot [6] registered [ 0.246127] acpiphp: Slot [7] registered [ 0.248116] acpiphp: Slot [8] registered [ 0.249116] acpiphp: Slot [9] registered [ 0.251143] acpiphp: Slot [10] registered [ 0.252125] acpiphp: Slot [3] registered [ 0.254114] acpiphp: Slot [4] registered [ 0.255102] acpiphp: Slot [11] registered [ 0.257114] acpiphp: Slot [12] registered [ 0.259101] acpiphp: Slot [13] registered [ 0.260102] acpiphp: Slot [14] registered [ 0.262116] acpiphp: Slot [15] registered [ 0.263100] acpiphp: Slot [16] registered [ 0.264093] acpiphp: Slot [17] registered [ 0.266099] acpiphp: Slot [18] registered [ 0.267089] acpiphp: Slot [19] registered [ 0.269116] acpiphp: Slot [20] registered [ 0.271158] acpiphp: Slot [21] registered [ 0.272120] acpiphp: Slot [22] registered [ 0.274109] acpiphp: Slot [23] registered [ 0.275099] acpiphp: Slot [24] registered [ 0.277117] acpiphp: Slot [25] registered [ 0.278102] acpiphp: Slot [26] registered [ 0.280127] acpiphp: Slot [27] registered [ 0.282099] acpiphp: Slot [28] registered [ 0.283104] acpiphp: Slot [29] registered [ 0.285102] acpiphp: Slot [30] registered [ 0.286165] acpiphp: Slot [31] registered [ 0.288064] PCI host bridge to bus 0000:00 [ 0.289020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307948] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325905] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336553] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338745] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344030] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372948] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476975] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.532032] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.544032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.553014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.617015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.631047] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.638020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685966] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.688449] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690331] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.696321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.700098] iommu: Default domain type: Passthrough [ 0.703488] SCSI subsystem initialized [ 0.705169] ACPI: bus type USB registered [ 0.706092] usbcore: registered new interface driver usbfs [ 0.708125] usbcore: registered new interface driver hub [ 0.710079] usbcore: registered new device driver usb [ 0.712192] pps_core: LinuxPPS API ver. 1 registered [ 0.714012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.717064] PTP clock support registered [ 0.719135] EDAC MC: Ver: 3.0.0 [ 0.721147] PCI: Using ACPI for IRQ routing [ 0.722913] NetLabel: Initializing [ 0.724011] NetLabel: domain hash size = 128 [ 0.726011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.728082] NetLabel: unlabeled traffic allowed by default [ 0.731121] vgaarb: loaded [ 0.732251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.733006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.739165] clocksource: Switched to clocksource kvm-clock [ 0.832263] VFS: Disk quotas dquot_6.6.0 [ 0.834376] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836355] *** VALIDATE ramfs *** [ 0.837375] *** VALIDATE hugetlbfs *** [ 0.838347] pnp: PnP ACPI init [ 0.839857] pnp: PnP ACPI: found 6 devices [ 0.853816] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855964] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857447] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858784] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860857] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862590] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.865279] NET: Registered protocol family 2 [ 0.867115] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.870618] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873043] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.877377] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.880036] TCP: Hash tables configured (established 65536 bind 65536) [ 0.881961] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.883932] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885900] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887858] NET: Registered protocol family 1 [ 0.889865] RPC: Registered named UNIX socket transport module. [ 0.891939] RPC: Registered udp transport module. [ 0.893784] RPC: Registered tcp transport module. [ 0.895672] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.898307] NET: Registered protocol family 44 [ 0.900044] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902304] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.904613] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.907245] PCI: CLS 0 bytes, default 64 [ 0.909491] Unpacking initramfs... [ 2.285827] debug: unmapping init [mem 0xffff8a2d3cc54000-0xffff8a2d3ffbffff] [ 2.289809] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.292104] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.295180] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.821523] Initialise system trusted keyrings [ 2.823283] Key type blacklist registered [ 2.825274] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.834329] zbud: loaded [ 2.837367] *** VALIDATE nfs *** [ 2.838673] *** VALIDATE nfs4 *** [ 2.840476] pstore: using deflate compression [ 2.844076] Platform Keyring initialized [ 2.944188] NET: Registered protocol family 38 [ 2.946701] Key type asymmetric registered [ 2.948685] Asymmetric key parser 'x509' registered [ 2.950592] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.955327] io scheduler mq-deadline registered [ 2.957587] io scheduler kyber registered [ 2.959401] io scheduler bfq registered [ 2.961423] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.964239] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.966996] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.970400] ACPI: Power Button [PWRF] [ 2.975667] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.983108] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.004431] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.012537] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.027683] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.057024] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.086667] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.091742] Non-volatile memory driver v1.3 [ 3.093497] Linux agpgart interface v0.103 [ 3.127255] virtio_blk virtio1: [vda] 146728 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.130344] vda: detected capacity change from 0 to 75124736 [ 3.149677] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.152750] vdb: detected capacity change from 0 to 1073741824 [ 3.167688] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.170640] vdc: detected capacity change from 0 to 2621440000 [ 3.188036] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.191622] vdd: detected capacity change from 0 to 2621440000 [ 3.210106] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.213869] vde: detected capacity change from 0 to 4294967296 [ 3.233433] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.235685] vdf: detected capacity change from 0 to 4294967296 [ 3.249118] libphy: Fixed MDIO Bus: probed [ 3.258128] usbcore: registered new interface driver usbserial_generic [ 3.261036] usbserial: USB Serial support registered for generic [ 3.264070] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.268378] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.270213] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.273044] mousedev: PS/2 mouse device common for all mice [ 3.276536] rtc_cmos 00:05: RTC can wake from S4 [ 3.279349] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.281246] rtc_cmos 00:05: registered as rtc0 [ 3.285903] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.286956] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.289280] intel_pstate: CPU model not supported [ 3.296253] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.296616] hid: raw HID events driver (C) Jiri Kosina [ 3.302049] usbcore: registered new interface driver usbhid [ 3.304349] usbhid: USB HID core driver [ 3.306076] drop_monitor: Initializing network drop monitor service [ 3.308723] Initializing XFRM netlink socket [ 3.311332] NET: Registered protocol family 10 [ 3.314394] Segment Routing with IPv6 [ 3.316260] NET: Registered protocol family 17 [ 3.318814] mpls_gso: MPLS GSO support [ 3.324340] RAS: Correctable Errors collector initialized. [ 3.326321] AVX version of gcm_enc/dec engaged. [ 3.327839] AES CTR mode by8 optimization enabled [ 3.403820] sched_clock: Marking stable (3403790325, 0)->(4272866932, -869076607) [ 3.406715] registered taskstats version 1 [ 3.408585] Loading compiled-in X.509 certificates [ 3.411042] zswap: loaded using pool lzo/zbud [ 3.434182] Key type big_key registered [ 3.447086] Key type encrypted registered [ 3.448332] ima: No TPM chip found, activating TPM-bypass! [ 3.449718] ima: Allocated hash algorithm: sha1 [ 3.451128] ima: No architecture policies found [ 3.452515] evm: Initialising EVM extended attributes: [ 3.454255] evm: security.selinux [ 3.455421] evm: security.ima [ 3.456270] evm: security.capability [ 3.457270] evm: HMAC attrs: 0x1 [ 3.459337] rtc_cmos 00:05: setting system clock to 2026-09-08 03:04:54 UTC (1788836694) [ 3.465408] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 3.467935] debug: unmapping init [mem 0xffffffff89782000-0xffffffff89a58fff] [ 3.476184] Write protecting the kernel read-only data: 28672k [ 3.479318] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 3.481503] debug: unmapping init [mem 0xffffffff88714000-0xffffffff887fffff] [ 3.515233] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.521713] systemd[1]: Detected virtualization kvm. [ 3.523204] systemd[1]: Detected architecture x86-64. [ 3.524631] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.556481] systemd[1]: No hostname configured. [ 3.557884] systemd[1]: Set hostname to . [ 3.559786] random: systemd: uninitialized urandom read (16 bytes read) [ 3.561854] systemd[1]: Initializing machine ID from random generator. [ 3.615650] random: ln: uninitialized urandom read (6 bytes read) [ 3.717573] random: systemd: uninitialized urandom read (16 bytes read) [ 3.719962] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.725552] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.731494] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.298224] device-mapper: uevent: version 1.0.3 [ 4.300649] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.100994] random: fast init done [ 5.136258] virtio_net virtio0 ens2: renamed from eth0 [ 5.166420] scsi host0: ata_piix [ 5.198580] scsi host1: ata_piix [ 5.200136] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.202719] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.045933] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.961969] random: crng init done [ 9.963427] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.489488] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.693522] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.922380] SELinux: Disabled at runtime. [ 11.988820] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.997408] systemd[1]: Detected virtualization kvm. [ 11.999362] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.478790] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.482102] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.488442] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.492498] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.496216] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.503483] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.508166] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 12.664055] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.003623] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.285458] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.315199] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.436307] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.449555] EDAC sbridge: Ver: 1.1.2 [ 15.334865] Key type dns_resolver registered [ 15.645251] NFS: Registering the id_resolver key type [ 15.647356] Key type id_resolver registered [ 15.648761] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg213-server login: [ 69.347101] libcfs: loading out-of-tree module taints kernel. [ 69.406981] Key type ._llcrypt registered [ 69.411471] Key type .llcrypt registered [ 69.584867] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 94.055399] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 95.810877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.837177] alg: No test for adler32 (adler32-zlib) [ 97.376646] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 98.426822] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 100.208254] Key type lgssc registered [ 102.278575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.042020] hrtimer: interrupt took 4010845 ns [ 123.993912] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.870254] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 185.596943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.657799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.054082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.116816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.223441] Lustre: lustre-MDT0000: new disk, initializing [ 187.354064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.403371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.692263] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.268917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.425818] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.459896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.462400] Lustre: Skipped 1 previous similar message [ 208.524777] Lustre: lustre-MDT0001: new disk, initializing [ 208.627461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.689163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.701823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.118797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.586721] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.366228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.726774] Lustre: lustre-OST0000: new disk, initializing [ 232.739449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.757663] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.904389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.721786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.729273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.847183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.111714] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.247524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.506312] Lustre: lustre-OST0001: new disk, initializing [ 258.512659] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.521798] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.599185] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.772511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 263.786173] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.915283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.366061] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.223169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.281870] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.621610] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.067846] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing yml_node [ 304.451057] Lustre: DEBUG MARKER: Client: 2.17.57.113 [ 307.853949] Lustre: DEBUG MARKER: MDS: 2.17.57.113 [ 310.679996] Lustre: DEBUG MARKER: OSS: 2.17.57.113 [ 312.343588] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 23:10:02 EDT 2026 [ 332.811476] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 334.342827] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.201357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.205539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.218089] Lustre: Skipped 3 previous similar messages [ 351.238819] Lustre: Skipped 3 previous similar messages [ 356.321073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.332154] Lustre: Skipped 3 previous similar messages [ 357.299107] Lustre: server umount lustre-MDT0000 complete [ 361.053469] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837052 with bad export cookie 11337482836302429076 [ 361.060899] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.062451] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 361.445613] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.453663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.456045] Lustre: Skipped 1 previous similar message [ 361.476606] Lustre: Skipped 1 previous similar message [ 366.562900] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.563813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.576677] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 366.603495] Lustre: Skipped 1 previous similar message [ 367.476259] Lustre: server umount lustre-MDT0001 complete [ 371.737464] Lustre: server umount lustre-OST0000 complete [ 375.798937] Lustre: server umount lustre-OST0001 complete [ 383.804537] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 392.864684] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 428.516326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.754042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.778456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.854980] Lustre: lustre-MDT0000: new disk, initializing [ 428.949413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.969573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.986170] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 447.315529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.504953] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 447.571051] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 447.577484] Lustre: Skipped 1 previous similar message [ 447.659040] Lustre: lustre-MDT0001: new disk, initializing [ 447.801100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.859037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.873325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.178839] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 465.410535] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.418805] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.844689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.367630] Lustre: lustre-OST0000: new disk, initializing [ 479.371398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.379835] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.555836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.294853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.302671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.481652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.949248] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 496.731850] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 501.555907] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 501.880310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.295466] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.465789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.969632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.981332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.007857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.482380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.662596] Lustre: server umount lustre-OST0000 complete [ 522.731387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.751265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 522.761958] Lustre: Skipped 2 previous similar messages [ 522.805321] Lustre: Skipped 3 previous similar messages [ 526.420850] Lustre: server umount lustre-MDT0000 complete [ 527.846457] LustreError: 15944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.872586] LustreError: 15944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 530.705660] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837221 with bad export cookie 11337482836302430609 [ 530.708327] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.725156] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 531.188804] Lustre: server umount lustre-MDT0001 complete [ 535.944507] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:13:46 (1788837226) [ 538.911188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 540.724563] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:13:50 (1788837230) [ 546.769870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.339374] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.456995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.320414] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 552.929596] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.030279] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.399437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.879950] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 575.405454] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.049733] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.606072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.290436] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 603.702778] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 616.931023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.937863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.950551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.976926] Lustre: Skipped 1 previous similar message [ 620.596217] Lustre: server umount lustre-OST0000 complete [ 627.169747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.727407] Lustre: server umount lustre-MDT0000 complete [ 632.296312] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.319732] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 634.286400] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837325 with bad export cookie 11337482836302431484 [ 634.295553] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.548551] Lustre: server umount lustre-MDT0001 complete [ 645.852618] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 648.393795] Key type lgssc unregistered [ 648.755931] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.764966] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.789792] LNet: Removed LNI 192.168.202.113@tcp [ 649.877512] Key type .llcrypt unregistered [ 649.885150] Key type ._llcrypt unregistered [ 671.973882] Key type ._llcrypt registered [ 671.975065] Key type .llcrypt registered [ 672.093523] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 673.090486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.176118] alg: No test for adler32 (adler32-zlib) [ 674.239217] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 674.518216] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 676.184725] Key type lgssc registered [ 677.414574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 685.935358] loop: module loaded [ 686.039712] loop0: detected capacity change from 0 to 204800000 [ 686.283582] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.303336] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.746668] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.768971] loop0: detected capacity change from 0 to 204800000 [ 693.012168] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.022605] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.045167] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.319903] loop0: detected capacity change from 0 to 204800000 [ 703.894870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.914821] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.080996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 705.158931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 705.279975] Lustre: lustre-MDT0000: new disk, initializing [ 705.416019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.441687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 709.951414] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 718.299301] loop1: detected capacity change from 0 to 204800000 [ 725.520672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.123846] Lustre: lustre-OST07c6: new disk, initializing [ 726.134937] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 726.141932] Lustre: Skipped 1 previous similar message [ 726.145066] Lustre: 26179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 726.371650] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 734.009921] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 734.883030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 734.899625] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 734.958583] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 739.541834] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 739.549886] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 739.557806] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 739.564408] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 739.569308] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 739.576210] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 754.238185] Lustre: 26844:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 754.261539] Lustre: 26844:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.276658] Lustre: 26844:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.287204] Lustre: 26844:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 754.294088] Lustre: 26844:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 754.303684] Lustre: 26844:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.194621] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 759.199286] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 759.204691] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 759.208254] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.211493] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.218889] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.224389] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 759.228758] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.234689] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.240573] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.249389] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.256889] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.462516] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.473143] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.482434] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.490985] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.501798] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.508564] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.518408] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.533057] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.541270] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.546277] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.548868] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.557036] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.935107] Lustre: 26844:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 766.941220] Lustre: 26844:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 766.952932] Lustre: 26844:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.961928] Lustre: 26844:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.977286] Lustre: 26844:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.984664] Lustre: 26844:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.993703] Lustre: 26844:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 767.003424] Lustre: 26844:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.017668] Lustre: 26844:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 767.029522] Lustre: 26844:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.037471] Lustre: 26844:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.043785] Lustre: 26844:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.917974] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 777.927263] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 777.936546] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.945996] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.955117] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.967156] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.975232] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.983230] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.991714] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 778.006823] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 778.013330] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.018716] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 826.867166] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.893053] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 831.380722] Lustre: server umount lustre-OST07c6 complete [ 841.926600] Lustre: server umount lustre-MDT0000 complete [ 861.366912] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 863.956705] Key type lgssc unregistered [ 864.136755] LNet: 28735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.144885] LNetError: 28735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.154245] LNet: Removed LNI 192.168.202.113@tcp [ 864.908224] Key type .llcrypt unregistered [ 864.912152] Key type ._llcrypt unregistered [ 883.587312] Key type ._llcrypt registered [ 883.589913] Key type .llcrypt registered [ 883.719270] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 884.509498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.621714] alg: No test for adler32 (adler32-zlib) [ 885.625076] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 886.006300] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 887.721434] Key type lgssc registered [ 889.351525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 897.723446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 897.740658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.334691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.202964] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 909.798891] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.840377] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 914.737147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.999136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 919.206770] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 928.247737] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.912757] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 942.118631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 942.649853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.700426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 949.145895] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 956.465994] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 974.305729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 974.313060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.317978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.321206] Lustre: Skipped 1 previous similar message [ 977.003840] Lustre: server umount lustre-OST0000 complete [ 981.478257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 981.495773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.516243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.518299] Lustre: Skipped 1 previous similar message [ 984.546090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.549457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.561846] Lustre: Skipped 1 previous similar message [ 989.673605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.687372] Lustre: Skipped 1 previous similar message [ 994.797574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.803654] Lustre: Skipped 1 previous similar message [ 995.298070] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 995.505538] Lustre: server umount lustre-MDT0000 complete [ 999.381592] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837690 with bad export cookie 18164817072832696113 [ 999.393864] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.399715] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.950502] Lustre: server umount lustre-MDT0001 complete [ 1010.652823] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1013.227650] Key type lgssc unregistered [ 1013.514174] LNet: 33761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.524611] LNetError: 33761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.545708] LNet: Removed LNI 192.168.202.113@tcp [ 1014.389511] Key type .llcrypt unregistered [ 1014.392182] Key type ._llcrypt unregistered [ 1031.045668] Key type ._llcrypt registered [ 1031.048263] Key type .llcrypt registered [ 1031.181716] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1032.291724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.372812] alg: No test for adler32 (adler32-zlib) [ 1033.619770] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1033.969067] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1035.746901] Key type lgssc registered [ 1037.887904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1048.514064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1048.549547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.035382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.943732] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1060.329042] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.347131] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1064.083629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.472151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.834953] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1077.892859] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1083.470631] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.255622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.578680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.735671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1096.357769] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1103.433403] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.203230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.208498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.215392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1123.660939] Lustre: server umount lustre-OST0000 complete [ 1129.453030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.465140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.480367] Lustre: Skipped 2 previous similar messages [ 1129.500900] Lustre: Skipped 2 previous similar messages [ 1134.534262] Lustre: server umount lustre-MDT0000 complete [ 1134.564349] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.581039] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1138.648184] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837829 with bad export cookie 16671879770932842861 [ 1138.652165] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.665372] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1139.104698] Lustre: server umount lustre-MDT0001 complete [ 1152.016951] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1155.417843] Key type lgssc unregistered [ 1155.730835] LNet: 38562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.744093] LNetError: 38562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.764601] LNet: Removed LNI 192.168.202.113@tcp [ 1156.974098] Key type .llcrypt unregistered [ 1156.976914] Key type ._llcrypt unregistered [ 1182.536873] Key type ._llcrypt registered [ 1182.539992] Key type .llcrypt registered [ 1182.670848] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1183.806159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1183.825382] alg: No test for adler32 (adler32-zlib) [ 1185.045054] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1185.401662] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1187.208212] Key type lgssc registered [ 1188.783299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.176834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.199486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.628342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.716917] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1207.780240] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.809418] LustreError: 39977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1210.139709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.731639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.119706] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1225.686311] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1232.119360] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1239.013972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1239.469659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.542608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1246.457797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1255.041099] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1262.051720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1262.067245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.082975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.326899] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1266.154766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.156136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.430044] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1268.444717] Lustre: Skipped 2 previous similar messages [ 1273.539616] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1273.549064] Lustre: Skipped 3 previous similar messages [ 1276.384594] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1276.607459] Lustre: server umount lustre-MDT0000 complete [ 1278.657428] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.228807] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837972 with bad export cookie 7717182228762435513 [ 1281.232470] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1281.239134] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1281.505917] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.507940] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.559745] Lustre: Skipped 1 previous similar message [ 1281.565665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1281.570473] Lustre: Skipped 3 previous similar messages [ 1283.769432] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.716711] Lustre: server umount lustre-MDT0001 complete [ 1303.283956] Lustre: server umount lustre-OST0000 complete [ 1322.790414] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1325.295908] Key type lgssc unregistered [ 1325.606902] LNet: 43838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.612995] LNetError: 43838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.634836] LNet: Removed LNI 192.168.202.113@tcp [ 1326.674181] Key type .llcrypt unregistered [ 1326.677399] Key type ._llcrypt unregistered [ 1353.459741] Key type ._llcrypt registered [ 1353.464181] Key type .llcrypt registered [ 1353.596351] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1355.338969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.403994] alg: No test for adler32 (adler32-zlib) [ 1356.643739] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1357.026557] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1358.776216] Key type lgssc registered [ 1360.312208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1368.961233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1368.983151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.674928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.958280] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1380.835475] LustreError: 45259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.868324] LustreError: 45259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1385.325109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.794852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.064731] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1399.275579] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.980544] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.100482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.438344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.497073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1421.065983] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1428.072582] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1440.002214] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1441.280793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.294118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.297516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.313841] Lustre: Skipped 1 previous similar message [ 1443.325565] Lustre: server umount lustre-OST0000 complete [ 1462.753881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.775625] LustreError: Skipped 1 previous similar message [ 1462.789826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.808369] Lustre: Skipped 1 previous similar message [ 1462.817903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.853361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.856100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.888639] Lustre: Skipped 1 previous similar message [ 1468.566815] Lustre: server umount lustre-MDT0000 complete [ 1471.973273] LustreError: 46549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.989522] LustreError: 46549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1472.220580] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838163 with bad export cookie 3863379419747814761 [ 1472.231140] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1472.236912] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.799405] Lustre: server umount lustre-MDT0001 complete [ 1491.078741] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1494.544739] Key type lgssc unregistered [ 1494.897839] LNet: 48966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.905394] LNetError: 48966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.914841] LNet: Removed LNI 192.168.202.113@tcp [ 1495.975552] Key type .llcrypt unregistered [ 1495.979122] Key type ._llcrypt unregistered [ 1520.801161] Key type ._llcrypt registered [ 1520.803366] Key type .llcrypt registered [ 1520.896530] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1522.480473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.527344] alg: No test for adler32 (adler32-zlib) [ 1523.651907] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1523.902938] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1525.568628] Key type lgssc registered [ 1526.855155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1535.177326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1535.195714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.691621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.027463] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1546.226153] LustreError: 50378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1546.261459] LustreError: 50378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1551.098763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.330078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1551.492237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.687462] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1564.691470] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.870699] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1578.344722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.830294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.938081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1585.590994] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1594.444518] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1602.109435] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1606.262982] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1618.666326] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0723h31m48s [ 1620.152772] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1622.750388] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1623.009766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1623.017558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.806828] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1623.840452] Lustre: Skipped 3 previous similar messages [ 1628.673782] Lustre: server umount lustre-MDT0000 complete [ 1628.857810] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.183331] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.198181] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1632.386344] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1635.013502] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.527457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.754576] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.013461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.085411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1641.079339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1645.684595] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1649.445525] Lustre: DEBUG MARKER: Wait for df (36697) ... [ 1651.651489] Lustre: DEBUG MARKER: done [ 1656.808582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.820764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.837661] Lustre: Skipped 3 previous similar messages [ 1656.863775] Lustre: Skipped 4 previous similar messages [ 1661.922943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.675459] Lustre: server umount lustre-OST0000 complete [ 1667.044110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1667.048454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.072436] Lustre: Skipped 1 previous similar message [ 1673.193697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.208068] Lustre: Skipped 6 previous similar messages [ 1681.378874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1681.823491] Lustre: server umount lustre-MDT0000 complete [ 1685.488826] LustreError: 50378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.507304] LustreError: 50378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1687.250659] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838378 with bad export cookie 7629237445426531979 [ 1687.288054] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.689095] Lustre: server umount lustre-MDT0001 complete [ 1700.460994] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1703.863847] Key type lgssc unregistered [ 1704.186233] LNet: 55306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.196764] LNetError: 55306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.213708] LNet: Removed LNI 192.168.202.113@tcp [ 1705.387759] Key type .llcrypt unregistered [ 1705.388874] Key type ._llcrypt unregistered [ 1753.875250] Key type ._llcrypt registered [ 1753.881211] Key type .llcrypt registered [ 1754.127220] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1755.533876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1755.663083] alg: No test for adler32 (adler32-zlib) [ 1756.806420] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1757.325029] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1759.073292] Key type lgssc registered [ 1760.916543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.231081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.254136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.582528] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1773.611814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1773.771375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.471547] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1791.662035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.868487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1791.905733] Lustre: 57516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1792.243366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.917844] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1807.355770] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1814.396709] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1822.826674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1823.169732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1823.462128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.869885] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1831.056845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1838.514542] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1846.380803] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0723h35m36s [ 1847.952512] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1855.796340] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1859.433430] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1859.548878] LustreError: 57525:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1879.825476] Lustre: lustre-MDT0000: Client 16d44fb8-46e7-4f29-bf16-53811049e05c (at 192.168.202.13@tcp) reconnecting [ 1899.569560] LustreError: 57525:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1902.036409] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1905.002714] Lustre: DEBUG MARKER: done [ 1912.807438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1912.816084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.833559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1913.321081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.344387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1913.348425] Lustre: Skipped 1 previous similar message [ 1918.437425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.446577] Lustre: Skipped 1 previous similar message [ 1918.715601] Lustre: server umount lustre-OST0000 complete [ 1923.559964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.577169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1923.597178] Lustre: Skipped 1 previous similar message [ 1929.691755] Lustre: server umount lustre-MDT0000 complete [ 1934.659189] LustreError: 57509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838625 with bad export cookie 11593278110128247337 [ 1934.669529] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.675675] LustreError: 57509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1935.219254] Lustre: server umount lustre-MDT0001 complete [ 1950.556120] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1954.632376] Key type lgssc unregistered [ 1954.940611] LNet: 61848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.950433] LNetError: 61848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.971262] LNet: Removed LNI 192.168.202.113@tcp [ 1955.839327] Key type .llcrypt unregistered [ 1955.841232] Key type ._llcrypt unregistered [ 2000.988575] Key type ._llcrypt registered [ 2000.990448] Key type .llcrypt registered [ 2001.103709] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2001.938955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.988084] alg: No test for adler32 (adler32-zlib) [ 2003.065553] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2003.486286] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2005.336182] Key type lgssc registered [ 2006.604519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.730558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.761366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.258845] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2012.295344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2012.477476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.813173] Lustre: Failing over lustre-MDT0000 [ 2014.752020] Lustre: server umount lustre-MDT0000 complete [ 2023.856583] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:38:33 (1788838713) [ 2032.214057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.886705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.303023] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2051.278569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.408031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2051.457221] Lustre: 64643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.816793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.123363] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2067.699058] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.331932] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2084.326351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2084.728815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2084.975434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.466101] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2093.746119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2100.654678] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2106.334786] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2114.530070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.532281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.552207] Lustre: Skipped 2 previous similar messages [ 2114.554501] Lustre: Skipped 1 previous similar message [ 2119.553340] Lustre: server umount lustre-MDT0000 complete [ 2124.034497] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838815 with bad export cookie 1538978694593830738 [ 2124.042623] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.048306] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2124.475831] Lustre: server umount lustre-MDT0001 complete [ 2128.703353] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2134.394431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2140.129054] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838815/real 1788838815] req@000000001114e7aa x1875733322442240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.185484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.093019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.250451] Lustre: 63564:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838820/real 1788838820] req@00000000dbd2f2f6 x1875733322442368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.475566] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x155b8cf80828e835 [ 2149.482684] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2149.821831] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.839371] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2149.917530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.008144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2150.881416] Lustre: 63565:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838825/real 1788838825] req@000000009b2e794e x1875733322442624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2150.954250] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.973407] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2154.940404] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2155.040315] Lustre: 63563:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838830/real 1788838830] req@00000000e3968ea3 x1875733322442752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.002807] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.029404] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2159.010452] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.024207] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2164.214462] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.238876] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2164.514498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.915256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.011956] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2169.989740] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2178.493753] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2184.533786] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.927380] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.942326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.950350] Lustre: Skipped 2 previous similar messages [ 2194.966812] Lustre: Skipped 2 previous similar messages [ 2197.399302] Lustre: server umount lustre-MDT0000 complete [ 2200.040767] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.053316] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2201.075094] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838892 with bad export cookie 1538978694593833013 [ 2201.088072] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2201.091592] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.442447] Lustre: server umount lustre-MDT0001 complete [ 2207.146211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.691381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.537703] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838896/real 1788838896] req@000000000c2d70df x1875733322470656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.550345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.661163] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x155b8cf80828ed98 [ 2226.675265] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2226.692741] Lustre: Skipped 1 previous similar message [ 2226.960975] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.103417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.145270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2231.291598] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2232.164573] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838907/real 1788838907] req@0000000091df5fe6 x1875733322471296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788838923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2232.188286] Lustre: 63562:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2238.542036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.015535] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2244.077708] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2249.427944] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.232399] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2256.437470] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2262.968244] Lustre: server umount lustre-MDT0000 complete [ 2264.547159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.556752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.567565] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.579232] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2266.794945] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838957 with bad export cookie 1538978694593834392 [ 2266.814026] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.205220] Lustre: server umount lustre-MDT0001 complete [ 2274.146484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2278.002543] Lustre: DEBUG MARKER: files compared the same [ 2289.283552] Lustre: server umount lustre-OST0000 complete [ 2306.403833] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2309.369586] Key type lgssc unregistered [ 2309.785504] LNet: 74375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.795653] LNetError: 74375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.807372] LNet: Removed LNI 192.168.202.113@tcp [ 2310.531892] Key type .llcrypt unregistered [ 2310.534278] Key type ._llcrypt unregistered [ 2330.306453] Key type ._llcrypt registered [ 2330.309596] Key type .llcrypt registered [ 2330.411595] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2331.471331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.633063] alg: No test for adler32 (adler32-zlib) [ 2332.700505] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2332.943197] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2334.568377] Key type lgssc registered [ 2335.877944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.742444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2342.754868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.143116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.011492] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug +malloc all [ 2354.671033] LustreError: 75765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.696459] LustreError: 75765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2358.097233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.329468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2362.426978] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug +malloc all [ 2369.435594] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2375.174300] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2382.763986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.228626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.361692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2390.441762] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug +malloc all [ 2397.803364] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2409.954276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.956747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.960246] LustreError: Skipped 1 previous similar message [ 2409.960856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.978703] Lustre: Skipped 1 previous similar message [ 2411.707335] Lustre: server umount lustre-OST0000 complete [ 2415.073400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.075623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.085736] Lustre: Skipped 1 previous similar message [ 2415.099326] Lustre: Skipped 2 previous similar messages [ 2420.193324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.209795] Lustre: Skipped 1 previous similar message [ 2421.244440] Lustre: server umount lustre-MDT0000 complete [ 2424.661345] LustreError: 75747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839115 with bad export cookie 7663001807551625604 [ 2424.667448] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.683805] LustreError: 75747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2424.912861] Lustre: server umount lustre-MDT0001 complete [ 2434.996499] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2437.874120] Key type lgssc unregistered [ 2438.149758] LNet: 79299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.165605] LNetError: 79299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.198310] LNet: Removed LNI 192.168.202.113@tcp [ 2439.098737] Key type .llcrypt unregistered [ 2439.103770] Key type ._llcrypt unregistered [ 2461.119726] Key type ._llcrypt registered [ 2461.123787] Key type .llcrypt registered [ 2461.251370] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2462.717409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.804069] alg: No test for adler32 (adler32-zlib) [ 2463.987900] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2464.375196] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2466.065706] Key type lgssc registered [ 2467.620849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2476.416693] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2476.430762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2491.936600] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2497.056399] LustreError: 80685:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 2523.808518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.996733] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2542.320900] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2552.756202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.999233] Lustre: *** cfs_fail_loc=706, val=0*** [ 2553.702042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.809203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2558.826983] LustreError: 80710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.868435] LustreError: 80710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2559.570278] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2564.075729] LustreError: 81771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.093179] LustreError: 81771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.192791] LustreError: 81772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.217104] LustreError: 81772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.485727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.879245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2575.298892] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2584.649962] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.916608] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.785402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.793158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.801411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.298295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.312838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.334628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.931982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.935994] Lustre: Skipped 2 previous similar messages [ 2606.064406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.083434] Lustre: Skipped 1 previous similar message [ 2608.100343] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2608.362678] Lustre: server umount lustre-OST0000 complete [ 2615.779444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2615.792824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.812867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.014708] Lustre: server umount lustre-MDT0000 complete [ 2621.409885] LustreError: 81776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.458104] LustreError: 81776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2623.254874] LustreError: 80692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839314 with bad export cookie 16703574106217881341 [ 2623.264344] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2623.267306] LustreError: 80692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2623.737582] Lustre: server umount lustre-MDT0001 complete [ 2640.344539] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2644.705422] Key type lgssc unregistered [ 2644.947816] LNet: 84301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2644.951820] LNetError: 84301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2644.972996] LNet: Removed LNI 192.168.202.113@tcp [ 2645.924317] Key type .llcrypt unregistered [ 2645.926795] Key type ._llcrypt unregistered [ 2677.947808] Key type ._llcrypt registered [ 2677.950398] Key type .llcrypt registered [ 2678.031869] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2679.473365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2679.648015] alg: No test for adler32 (adler32-zlib) [ 2681.003582] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2681.443264] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2683.233651] Key type lgssc registered [ 2684.844920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2693.814392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2693.849587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.742416] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2709.154475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.672702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.107137] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.144523] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2716.288127] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2719.211746] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.234383] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2724.321627] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.346322] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2725.782248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.068078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.496927] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.521125] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2733.156887] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2739.693993] LustreError: 86402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.717706] LustreError: 86402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2743.241940] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.251056] Lustre: MGS: Client b07570e8-7f85-4cc3-b6fb-1dcd5226bdae (at 0@lo) reconnecting [ 2743.263892] LustreError: 86418:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2743.575524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.654479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2748.815866] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2764.260121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.268506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.290264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2769.388790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2769.399043] Lustre: Skipped 1 previous similar message [ 2770.395328] Lustre: server umount lustre-OST0000 complete [ 2774.499044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.502035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2774.524658] Lustre: Skipped 2 previous similar messages [ 2774.534118] Lustre: Skipped 1 previous similar message [ 2779.617798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.631880] Lustre: Skipped 2 previous similar messages [ 2780.801744] Lustre: server umount lustre-MDT0000 complete [ 2784.737830] LustreError: 87949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.783127] LustreError: 87949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2785.489966] Lustre: server umount lustre-MDT0001 complete [ 2789.430810] Lustre: server umount MGS complete [ 2801.985829] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:51:32 (1788839492) [ 2833.263490] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2845.412702] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2885.820949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.024625] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2904.181690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.298219] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2904.347839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2904.410196] Lustre: lustre-MDT0001: new disk, initializing [ 2904.500316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.543878] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2904.562704] Lustre: 93594:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2904.592853] LustreError: 93594:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2909.171832] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2920.645255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.945825] Lustre: lustre-OST0000: new disk, initializing [ 2920.950141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2920.958080] Lustre: 94360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2921.048242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.230760] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2931.233667] Lustre: 94796:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006b97eb36 x1875734033376640/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788839638 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2938.731272] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2945.906628] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.919603] Lustre: MGS: Client 2caad11d-ebd6-49fc-86e1-1063d6edd162 (at 0@lo) reconnecting [ 2946.001263] Lustre: 92862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2946.018719] LustreError: 93580:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2946.032754] LustreError: 93580:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2946.064892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2946.144506] Lustre: lustre-MDT0000: new disk, initializing [ 2946.295294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.323144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2946.336709] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2947.364170] Lustre: 95429:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000072817932 x1875734033383424/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788839654 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2951.617106] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2952.550863] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2956.465117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2956.481466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2956.608735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2982.369303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.374396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.388060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.418794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.429621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.718593] Lustre: server umount lustre-OST0000 complete [ 2996.196636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.200321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2996.213655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.213664] Lustre: Skipped 3 previous similar messages [ 2999.259251] Lustre: server umount lustre-MDT0000 complete [ 3001.315903] LustreError: 93569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.332129] LustreError: 93569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3009.563418] Lustre: server umount lustre-MDT0001 complete [ 3013.114285] Lustre: server umount MGS complete [ 3024.503232] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:55:14 (1788839714) [ 3043.392575] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3046.132831] Key type lgssc unregistered [ 3046.593385] LNet: 97932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3046.606244] LNetError: 97932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3046.627764] LNet: Removed LNI 192.168.202.113@tcp [ 3047.398219] Key type .llcrypt unregistered [ 3047.400876] Key type ._llcrypt unregistered [ 3064.246219] Key type ._llcrypt registered [ 3064.247877] Key type .llcrypt registered [ 3064.358189] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3065.099170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3065.116596] alg: No test for adler32 (adler32-zlib) [ 3066.183786] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3066.436888] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3068.096243] Key type lgssc registered [ 3069.363891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3077.151557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3077.176837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.901062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.754359] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3088.357609] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.385344] LustreError: 99152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3092.260775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.603645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3097.012373] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3103.912870] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3109.406564] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3115.853097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.226430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.390141] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3123.451724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3130.722202] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3143.141831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3143.153562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.183045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.164869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3144.165823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.175255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.861981] Lustre: server umount lustre-OST0000 complete [ 3153.891302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3153.895460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.901107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.903611] Lustre: Skipped 1 previous similar message [ 3159.521676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.530350] Lustre: Skipped 3 previous similar messages [ 3164.128237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3164.349238] Lustre: server umount lustre-MDT0000 complete [ 3164.643884] LustreError: 99151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.672106] LustreError: 99151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3168.214982] LustreError: 99132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839859 with bad export cookie 2057557375066555989 [ 3168.219216] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.231752] LustreError: 99132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3168.700340] Lustre: server umount lustre-MDT0001 complete [ 3180.811185] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3183.675447] Key type lgssc unregistered [ 3183.934907] LNet: 102684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3183.940982] LNetError: 102684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3183.957403] LNet: Removed LNI 192.168.202.113@tcp [ 3184.763843] Key type .llcrypt unregistered [ 3184.766479] Key type ._llcrypt unregistered [ 3200.519375] Key type ._llcrypt registered [ 3200.522769] Key type .llcrypt registered [ 3200.714111] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3202.117245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3202.179934] alg: No test for adler32 (adler32-zlib) [ 3203.359913] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3203.563667] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3205.217300] Key type lgssc registered [ 3206.480882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3212.494902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3212.506591] LustreError: 103762:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3212.507870] LustreError: 103764:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3212.522203] LustreError: 103762:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3212.529792] LustreError: 103762:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3212.537202] LustreError: 103762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3212.554469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.045140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.386047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.770942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.631944] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3231.440523] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing lsmod [ 3234.286106] LustreError: 104969:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3234.299028] LustreError: 104972:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3234.301602] LustreError: 104969:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3234.307472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.308095] LustreError: 104969:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3234.315323] LustreError: 104969:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3234.592367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.790744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.791952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.813419] Lustre: Skipped 1 previous similar message [ 3236.823388] Lustre: Skipped 1 previous similar message [ 3241.955430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.077501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.088346] Lustre: Skipped 3 previous similar messages [ 3251.168333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3251.354767] Lustre: server umount lustre-MDT0000 complete [ 3252.198885] LustreError: 103797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.234941] LustreError: 103797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3256.107730] LustreError: 103779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839947 with bad export cookie 18265963830940668857 [ 3256.122538] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.123041] LustreError: 103779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3257.315532] LustreError: 103798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.343354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.387669] Lustre: Skipped 1 previous similar message [ 3257.397473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3257.411577] Lustre: Skipped 1 previous similar message [ 3257.443231] LustreError: 104233:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000586de1c6 x1875734580713216/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3258.038617] Lustre: server umount lustre-MDT0001 complete [ 3264.171905] Lustre: server umount lustre-OST0000 complete [ 3274.560153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.739090] LustreError: 106035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.954943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.385644] LustreError: 106036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.183986] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3285.496120] LustreError: 106035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.300044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.591479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3297.993152] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3305.052730] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3311.952387] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3318.788197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.152613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.196223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3325.607111] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3333.394119] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3346.918654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3346.932350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.946528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.962157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.978648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.707061] Lustre: server umount lustre-OST0000 complete [ 3359.203098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.211291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.225023] Lustre: Skipped 1 previous similar message [ 3359.239974] Lustre: Skipped 3 previous similar messages [ 3363.349834] Lustre: server umount lustre-MDT0000 complete [ 3364.324249] LustreError: 106031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.354972] LustreError: 106031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3367.690555] LustreError: 106015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840058 with bad export cookie 18265963830940669571 [ 3367.694964] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.702250] LustreError: 106015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.034224] Lustre: server umount lustre-MDT0001 complete [ 3380.545961] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3383.613237] Key type lgssc unregistered [ 3384.048784] LNet: 109605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.060232] LNetError: 109605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.076579] LNet: Removed LNI 192.168.202.113@tcp [ 3385.027658] Key type .llcrypt unregistered [ 3385.033418] Key type ._llcrypt unregistered [ 3408.873958] Key type ._llcrypt registered [ 3408.877425] Key type .llcrypt registered [ 3409.000390] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3410.096828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3410.215848] alg: No test for adler32 (adler32-zlib) [ 3411.316298] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3411.624650] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3413.304752] Key type lgssc registered [ 3414.475591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3421.906243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3421.932644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.508524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.130446] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3433.958030] LustreError: 111021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.993910] LustreError: 111021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3436.788110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.251781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3442.566775] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3450.004526] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3455.994465] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3462.676418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.046117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.152814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3468.795292] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3476.147208] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3486.525191] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3492.014442] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3497.957917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.971155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.975869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.997388] Lustre: Skipped 1 previous similar message [ 3503.088193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.106932] Lustre: Skipped 1 previous similar message [ 3503.489921] Lustre: server umount lustre-OST0000 complete [ 3508.708467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3508.714099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.728470] Lustre: Skipped 1 previous similar message [ 3508.737389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.316138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.320938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.904387] Lustre: server umount lustre-MDT0000 complete [ 3518.450428] LustreError: 112008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.480433] LustreError: 112008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3519.743447] LustreError: 111001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840210 with bad export cookie 5280744213864612477 [ 3519.746085] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3519.755958] LustreError: 111001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3520.228681] Lustre: server umount lustre-MDT0001 complete [ 3535.104917] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3538.847296] Key type lgssc unregistered [ 3539.136954] LNet: 114662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.157047] LNetError: 114662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.173706] LNet: Removed LNI 192.168.202.113@tcp [ 3540.319688] Key type .llcrypt unregistered [ 3540.321693] Key type ._llcrypt unregistered [ 3560.052850] Key type ._llcrypt registered [ 3560.055817] Key type .llcrypt registered [ 3560.231316] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3561.738724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.852993] alg: No test for adler32 (adler32-zlib) [ 3563.029405] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3563.393523] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3565.088185] Key type lgssc registered [ 3566.532637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3574.074712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3574.083266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.430207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.760402] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3585.000785] LustreError: 115879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.018469] LustreError: 115879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3590.040396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.118131] LustreError: 115878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.491826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3594.564902] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3601.975264] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3608.000532] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3615.170743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3615.568418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.737960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3622.635801] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3630.594754] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3643.892684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3643.895941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.923429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.413834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.417720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3645.417853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.424762] Lustre: Skipped 1 previous similar message [ 3649.826215] Lustre: server umount lustre-OST0000 complete [ 3655.656648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.674734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.127591] Lustre: server umount lustre-MDT0000 complete [ 3660.778099] LustreError: 115879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.789065] LustreError: 115879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3663.836734] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840354 with bad export cookie 10588355169725779924 [ 3663.848153] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.854247] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3664.192446] Lustre: server umount lustre-MDT0001 complete [ 3677.223815] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3680.295336] Key type lgssc unregistered [ 3680.588246] LNet: 119437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3680.602121] LNetError: 119437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3680.629634] LNet: Removed LNI 192.168.202.113@tcp [ 3681.675908] Key type .llcrypt unregistered [ 3681.678154] Key type ._llcrypt unregistered [ 3706.639783] Key type ._llcrypt registered [ 3706.642535] Key type .llcrypt registered [ 3706.758326] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3707.958516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3708.014614] alg: No test for adler32 (adler32-zlib) [ 3709.091910] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3709.383278] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3711.113241] Key type lgssc registered [ 3712.516161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3721.398678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3721.417336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.944967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.597975] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3732.462301] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.483994] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3737.574666] LustreError: 120853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.103790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.544244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.006589] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3752.562946] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3759.680484] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3766.616086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.078607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.146804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3773.783256] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3782.501975] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3789.031453] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3793.073481] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3793.081264] Lustre: Skipped 2 previous similar messages [ 3796.056486] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3796.123322] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3797.383401] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3797.394582] Lustre: Skipped 2 previous similar messages [ 3801.550834] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3801.564050] Lustre: Skipped 1 previous similar message [ 3806.397047] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3806.411506] Lustre: Skipped 2 previous similar messages [ 3810.353483] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.13@tcp 192.168.0.[2,10]@tcp [ 3815.612508] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.13/24@tcp in log lustre-MDT0000 [ 3815.617508] Lustre: Skipped 2 previous similar messages [ 3817.872397] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.13/24@tcp [ 3817.882654] Lustre: Skipped 1 previous similar message [ 3828.196833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3828.200081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.207738] LustreError: Skipped 2 previous similar messages [ 3828.233062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.731165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.731382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.734616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.734626] LustreError: Skipped 2 previous similar messages [ 3834.251518] Lustre: server umount lustre-OST0000 complete [ 3839.975705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.978445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.992872] Lustre: Skipped 1 previous similar message [ 3840.008675] Lustre: Skipped 2 previous similar messages [ 3844.853541] Lustre: server umount lustre-MDT0000 complete [ 3845.099363] LustreError: 124308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.136588] LustreError: 124308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3849.087874] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840540 with bad export cookie 3692838873648193807 [ 3849.091551] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.099221] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3849.504867] Lustre: server umount lustre-MDT0001 complete [ 3862.393909] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3865.536672] Key type lgssc unregistered [ 3865.978592] LNet: 125656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3865.988449] LNetError: 125656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.048429] LNet: Removed LNI 192.168.202.113@tcp [ 3868.240724] Key type .llcrypt unregistered [ 3868.252418] Key type ._llcrypt unregistered [ 3903.476510] Key type ._llcrypt registered [ 3903.479542] Key type .llcrypt registered [ 3903.599379] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3905.399146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3905.443798] alg: No test for adler32 (adler32-zlib) [ 3906.710870] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 3907.057679] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3908.832226] Key type lgssc registered [ 3910.055288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.050558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3918.065693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.624032] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3919.667311] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.13/24@tcp [ 3919.676358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.625389] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3929.070496] LustreError: 127217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.079657] LustreError: 127217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3934.177529] LustreError: 127216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.275318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.709738] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3934.761921] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.13/24@tcp [ 3934.806674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.679458] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3947.723357] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3953.909357] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3961.235116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3961.727957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.788163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3968.867912] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3977.770287] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3988.982213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3988.991279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.014422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.518630] Lustre: server umount lustre-OST0000 complete [ 3999.715367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.721940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.729472] Lustre: Skipped 2 previous similar messages [ 3999.741848] Lustre: Skipped 3 previous similar messages [ 4004.795841] Lustre: server umount lustre-MDT0000 complete [ 4004.840417] LustreError: 127216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.867579] LustreError: 127216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4009.858589] LustreError: 127196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840700 with bad export cookie 2077733360679672694 [ 4009.878553] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.254898] Lustre: server umount lustre-MDT0001 complete [ 4022.629694] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4025.478146] Key type lgssc unregistered [ 4025.900615] LNet: 130806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.910401] LNetError: 130806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4025.930473] LNet: Removed LNI 192.168.202.113@tcp [ 4026.810186] Key type .llcrypt unregistered [ 4026.811768] Key type ._llcrypt unregistered [ 4055.675829] Key type ._llcrypt registered [ 4055.677921] Key type .llcrypt registered [ 4055.812945] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4071.919402] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4072.874574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.094961] alg: No test for adler32 (adler32-zlib) [ 4074.090625] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 4074.300595] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4075.944185] Key type lgssc registered [ 4077.337183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4114.439114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4114.462423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.810608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4115.849254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4115.977151] Lustre: lustre-MDT0000: new disk, initializing [ 4116.083036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.100579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.842519] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4130.615651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.820464] Lustre: 134205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4130.870740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4130.878070] Lustre: Skipped 1 previous similar message [ 4130.957678] Lustre: lustre-MDT0001: new disk, initializing [ 4131.101474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4131.139368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4131.152895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4136.029684] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4147.487506] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4154.049874] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4161.156363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.376819] Lustre: lustre-OST0000: new disk, initializing [ 4161.380181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4161.387707] Lustre: 136173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4161.473557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.580491] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4170.828037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4170.842996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4171.022552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4178.147640] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4183.015065] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.228113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4187.999791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4188.268187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4191.715230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.718121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.744022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.631823] Lustre: server umount lustre-OST0000 complete [ 4201.955906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.960695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.965705] Lustre: Skipped 2 previous similar messages [ 4201.973874] Lustre: Skipped 2 previous similar messages [ 4206.670581] Lustre: server umount lustre-MDT0000 complete [ 4210.505858] LustreError: 134198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840901 with bad export cookie 2062613670225065189 [ 4210.514690] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4210.960892] Lustre: server umount lustre-MDT0001 complete [ 4220.671084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.151911] LustreError: 138389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.176187] LustreError: 138389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4221.242498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.530295] LustreError: 138390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.640182] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4231.654531] LustreError: 138389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.777584] LustreError: 138390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.221441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.605928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.806841] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4253.342164] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4259.760771] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4268.289533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4268.673622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.568887] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4282.589538] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4287.768739] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4292.451523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4297.412723] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4297.708298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4305.575875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4305.784519] Lustre: lustre-OST0001: new disk, initializing [ 4305.788351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4305.794112] Lustre: 141791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4305.873533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.699409] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4314.251990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4314.267483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4314.450664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4322.400090] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4322.664873] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4328.332847] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4328.574181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4334.927345] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff993dc6592800.ost_server_uuid 50 [ 4336.876018] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff993dc6592800.ost_server_uuid in IDLE state after 0 sec [ 4345.315556] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4345.330960] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.345155] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4345.347582] Lustre: Skipped 1 previous similar message [ 4345.364676] Lustre: Skipped 1 previous similar message [ 4346.450467] Lustre: server umount lustre-OST0001 complete [ 4350.447794] LustreError: 140212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.459392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.478907] LustreError: 140212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4350.495397] Lustre: Skipped 1 previous similar message [ 4350.499459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.528824] Lustre: Skipped 1 previous similar message [ 4355.562678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4355.566916] Lustre: Skipped 1 previous similar message [ 4356.699566] Lustre: server umount lustre-OST0000 complete [ 4365.798126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4365.802572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.812802] LustreError: Skipped 1 previous similar message [ 4365.827545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.846075] Lustre: Skipped 1 previous similar message [ 4367.272620] Lustre: server umount lustre-MDT0000 complete [ 4370.918493] LustreError: 138390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.936335] LustreError: 138390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4371.077325] LustreError: 138369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841062 with bad export cookie 2062613670225066071 [ 4371.079666] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.086822] LustreError: 138369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4371.366380] Lustre: server umount lustre-MDT0001 complete [ 4391.432405] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4394.388102] Key type lgssc unregistered [ 4394.666402] LNet: 144795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4394.678180] LNetError: 144795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.705581] LNet: Removed LNI 192.168.202.113@tcp [ 4396.098545] Key type .llcrypt unregistered [ 4396.102577] Key type ._llcrypt unregistered [ 4439.957421] Key type ._llcrypt registered [ 4439.963172] Key type .llcrypt registered [ 4440.081574] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4457.183144] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4458.386451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4458.409628] alg: No test for adler32 (adler32-zlib) [ 4459.500228] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 4459.774545] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4461.480988] Key type lgssc registered [ 4462.745600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4495.932553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4495.958781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.170754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4497.210264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4497.308424] Lustre: lustre-MDT0000: new disk, initializing [ 4497.451418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.479401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.935343] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4512.506958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.620294] Lustre: 148945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.687319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4512.690754] Lustre: Skipped 1 previous similar message [ 4512.746646] Lustre: lustre-MDT0001: new disk, initializing [ 4512.847201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4512.909835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4512.925984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.793122] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4527.166226] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4533.256425] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4540.523345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.883629] Lustre: lustre-OST0000: new disk, initializing [ 4540.887482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4540.892650] Lustre: 150915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4540.972885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.189893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4546.210838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4546.375862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4549.640422] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4559.473337] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4563.701544] Lustre: Failing over lustre-OST0000 [ 4565.852439] Lustre: server umount lustre-OST0000 complete [ 4582.368184] Lustre: 147171:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841257/real 1788841257] req@000000003998ca12 x1875735897736704/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788841273 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4582.402933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.398712] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841258/real 1788841258] req@00000000112f13cf x1875735897736960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788841274 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4583.438434] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4583.451206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.981745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.246904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.259786] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4588.256256] Lustre: 147171:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841263/real 1788841263] req@0000000087d68b50 x1875735897738752/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788841279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4591.812883] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4592.383785] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4593.376778] Lustre: 147171:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841268/real 1788841268] req@0000000047e8f584 x1875735897739904/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788841284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4593.414939] Lustre: 147171:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4593.462221] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4593.462453] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4597.020037] Lustre: Failing over lustre-MDT0000 [ 4597.335608] Lustre: server umount lustre-MDT0000 complete [ 4597.417641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.438571] Lustre: Skipped 2 previous similar messages [ 4598.048154] Lustre: 147172:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841273/real 1788841273] req@00000000a0b40d7a x1875735897740928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788841289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4598.073505] Lustre: 147172:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4602.849739] LustreError: 149780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.877355] LustreError: 149780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4607.169291] LustreError: 148954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.282248] LustreError: 148952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.307331] LustreError: 148952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4617.384187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.405580] LustreError: 148952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.432237] LustreError: 148952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4617.620362] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.102965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.231519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4622.526417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4622.927244] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4623.345066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4623.359979] Lustre: Skipped 1 previous similar message [ 4623.376605] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4623.415568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4633.579808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.595473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4637.099067] Lustre: server umount lustre-OST0000 complete [ 4643.809790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4643.813456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.819537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.829517] Lustre: Skipped 2 previous similar messages [ 4643.841671] Lustre: Skipped 2 previous similar messages [ 4646.895103] Lustre: server umount lustre-MDT0000 complete [ 4648.935562] LustreError: 148953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.959624] LustreError: 148953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4650.486591] LustreError: 148937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841341 with bad export cookie 6780308548945609158 [ 4650.497387] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4650.506573] LustreError: 148937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4650.953894] Lustre: server umount lustre-MDT0001 complete [ 4663.234950] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4666.166422] Key type lgssc unregistered [ 4666.420197] LNet: 154569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4666.425536] LNetError: 154569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4666.436189] LNet: Removed LNI 192.168.202.113@tcp [ 4667.178449] Key type .llcrypt unregistered [ 4667.180536] Key type ._llcrypt unregistered [ 4706.410746] Key type ._llcrypt registered [ 4706.418286] Key type .llcrypt registered [ 4706.631818] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4708.560971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4708.602412] alg: No test for adler32 (adler32-zlib) [ 4709.969592] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 4710.290213] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4711.960240] Key type lgssc registered [ 4713.391456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4723.023406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4723.064535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.432612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.148922] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4734.957822] LustreError: 156088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.971547] LustreError: 156088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4738.974268] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4740.080941] LustreError: 156089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4742.800891] LustreError: 156117:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4742.813163] LustreError: 156117:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4742.825170] LustreError: 156117:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4743.358792] Lustre: server umount lustre-MDT0000 complete [ 4759.059626] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4771.129453] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4810.734173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.895328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4810.925444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4811.026844] Lustre: lustre-MDT0000: new disk, initializing [ 4811.127945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.145874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4815.170491] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4826.757517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.915865] Lustre: 159422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4826.990166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4827.000641] Lustre: Skipped 1 previous similar message [ 4827.144384] Lustre: lustre-MDT0001: new disk, initializing [ 4827.456392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4827.608122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4827.630573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4834.230840] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4844.921380] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4852.792443] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4860.222408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.594717] Lustre: lustre-OST0000: new disk, initializing [ 4860.603788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4860.614593] Lustre: 161391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4860.791652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4867.829564] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4868.155467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4868.167949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4868.289041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4877.051381] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4882.385787] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4884.024574] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:26:14 (1788841574) [ 5160.417053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.426111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.438367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5160.449186] Lustre: Skipped 2 previous similar messages [ 5160.480979] Lustre: Skipped 2 previous similar messages [ 5164.938943] Lustre: server umount lustre-MDT0000 complete [ 5165.545777] LustreError: 159433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.566493] LustreError: 159433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5168.812834] LustreError: 161401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841859 with bad export cookie 3871731101104680312 [ 5168.813286] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.826072] LustreError: 161401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5170.657552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.660594] LustreError: 159427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.670857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5173.664899] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5175.402702] Lustre: server umount lustre-MDT0001 complete [ 5186.546504] Lustre: server umount lustre-OST0000 complete [ 5198.210808] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5211.377867] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5253.916050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.181084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5254.213587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5254.323936] Lustre: lustre-MDT0000: new disk, initializing [ 5254.460587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.492715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5260.811902] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5274.346046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.521527] Lustre: 166069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5274.597494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5274.604150] Lustre: Skipped 1 previous similar message [ 5274.726283] Lustre: lustre-MDT0001: new disk, initializing [ 5274.911283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.982568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5275.002311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5279.016698] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5288.798122] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5294.774514] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5301.605583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.820673] Lustre: lustre-OST0000: new disk, initializing [ 5301.824973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5301.832935] Lustre: 168036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5301.898315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.893980] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5307.967383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5307.982236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5308.113957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5319.245759] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5324.573457] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5324.819837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5330.703718] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5330.987437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5333.989886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.999752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.004234] LustreError: Skipped 1 previous similar message [ 5334.006211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.053628] Lustre: Skipped 1 previous similar message [ 5339.112976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.123934] Lustre: Skipped 1 previous similar message [ 5347.296248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5347.474814] Lustre: server umount lustre-OST0000 complete [ 5351.908784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5351.921927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.957251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.971878] Lustre: Skipped 3 previous similar messages [ 5358.031615] Lustre: server umount lustre-MDT0000 complete [ 5359.588230] LustreError: 166078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.600148] LustreError: 166078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5362.012304] LustreError: 166061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842053 with bad export cookie 3871731101104839142 [ 5362.019755] LustreError: 166061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5362.020311] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.309575] Lustre: server umount lustre-MDT0001 complete [ 5371.961417] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:34:21 (1788842061) [ 5390.715185] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5401.846841] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5442.257377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.540614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5442.559559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5442.646692] Lustre: lustre-MDT0000: new disk, initializing [ 5442.763504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.784676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5447.492625] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5460.058226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.208580] Lustre: 172796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5460.668260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5466.535903] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5479.402261] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5486.682648] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5494.090258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5494.336479] Lustre: lustre-OST0000: new disk, initializing [ 5494.339372] Lustre: Skipped 1 previous similar message [ 5494.342997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5494.347604] Lustre: Skipped 2 previous similar messages [ 5494.354578] Lustre: 174775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5500.993319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5501.007579] Lustre: Skipped 1 previous similar message [ 5501.015650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5501.181042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5502.521118] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5513.044137] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5521.889542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5521.899923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.915751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.923924] Lustre: Skipped 2 previous similar messages [ 5521.939310] Lustre: Skipped 3 previous similar messages [ 5535.200171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5535.418851] Lustre: server umount lustre-OST0000 complete [ 5542.372862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.570660] Lustre: server umount lustre-MDT0000 complete [ 5547.491273] LustreError: 172801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5547.510935] LustreError: 172801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5549.367440] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842240 with bad export cookie 3871731101104840024 [ 5549.372061] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5549.380039] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5549.901733] Lustre: server umount lustre-MDT0001 complete [ 5560.022438] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:37:30 (1788842250) [ 5583.317797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5595.852750] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5636.750607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.299971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5637.335195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5637.387443] Lustre: lustre-MDT0000: new disk, initializing [ 5637.449782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.452056] Lustre: Skipped 2 previous similar messages [ 5637.486151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5642.679703] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5653.803616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.018790] Lustre: 179104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5654.383240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5659.038551] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5670.862905] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5678.570035] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5685.650399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5686.082651] Lustre: 181081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5693.532958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5693.744936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5694.451652] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5705.389219] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5714.408268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.411970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.430749] Lustre: Skipped 1 previous similar message [ 5714.447944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.453079] LustreError: Skipped 1 previous similar message [ 5714.476956] Lustre: Skipped 7 previous similar messages [ 5719.965291] Lustre: server umount lustre-OST0000 complete [ 5730.511407] Lustre: server umount lustre-MDT0000 complete [ 5734.561827] LustreError: 179094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842425 with bad export cookie 3871731101104841109 [ 5734.567704] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.572197] LustreError: 179094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5734.967942] Lustre: server umount lustre-MDT0001 complete [ 5746.936762] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5750.046661] Key type lgssc unregistered [ 5750.367876] LNet: 182937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.376667] LNetError: 182937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.400678] LNet: Removed LNI 192.168.202.113@tcp [ 5751.671800] Key type .llcrypt unregistered [ 5751.674185] Key type ._llcrypt unregistered [ 5774.617830] Key type ._llcrypt registered [ 5774.625086] Key type .llcrypt registered [ 5774.715273] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5775.696975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.844118] alg: No test for adler32 (adler32-zlib) [ 5776.868120] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 5777.082697] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5778.728665] Key type lgssc registered [ 5779.751128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5787.071645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5787.102625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.591262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.299686] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5798.888505] LustreError: 184347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.906821] LustreError: 184347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5801.835887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.124634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5807.426649] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5814.882858] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5820.478276] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5827.370820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5827.783394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.910691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5834.789681] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5842.929651] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5864.930333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5864.937757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.968760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.003835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.014855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.021304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.030071] Lustre: Skipped 1 previous similar message [ 5870.695916] Lustre: server umount lustre-OST0000 complete [ 5878.243186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.261081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.267742] Lustre: Skipped 1 previous similar message [ 5880.591471] Lustre: server umount lustre-MDT0000 complete [ 5883.361617] LustreError: 184348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.390116] LustreError: 184348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5884.106043] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842575 with bad export cookie 18118658319833479907 [ 5884.118493] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.119454] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5884.444507] Lustre: server umount lustre-MDT0001 complete [ 5896.779508] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5899.585776] Key type lgssc unregistered [ 5899.853418] LNet: 187908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5899.871305] LNetError: 187908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5899.894797] LNet: Removed LNI 192.168.202.113@tcp [ 5900.791299] Key type .llcrypt unregistered [ 5900.793437] Key type ._llcrypt unregistered [ 5927.616534] Key type ._llcrypt registered [ 5927.618347] Key type .llcrypt registered [ 5927.796656] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5929.364187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.391682] alg: No test for adler32 (adler32-zlib) [ 5930.528941] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 5930.690315] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5932.336233] Key type lgssc registered [ 5933.337865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5941.309967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5941.336940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.742698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.334897] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5951.974096] LustreError: 189323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.994716] LustreError: 189323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5955.992286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.373448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.117629] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5969.453207] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5975.185323] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5983.186270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.458813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.575203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5989.452431] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5998.330256] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6007.278930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.279182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6007.299950] LustreError: 191144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.446954] Lustre: server umount lustre-OST0000 complete [ 6015.252145] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff993dc8b7c800.ost_server_uuid 50 [ 6028.748882] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6043.633228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.647440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.656935] Lustre: Skipped 1 previous similar message [ 6043.660296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.779217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.780213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.801538] Lustre: Skipped 1 previous similar message [ 6049.847102] Lustre: server umount lustre-MDT0000 complete [ 6052.837571] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.861078] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6054.358513] LustreError: 189303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842745 with bad export cookie 13409631262310694954 [ 6054.361427] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6054.363340] LustreError: 189303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6054.606950] Lustre: server umount lustre-MDT0001 complete [ 6065.795444] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:45:55 (1788842755) [ 6073.051301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.454950] LustreError: 193259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.529838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.659699] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6078.946585] LustreError: 193260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.163402] LustreError: 193259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.184578] LustreError: 193259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6088.486458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.891488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.426384] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6103.111304] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.987942] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6118.632448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.887650] LustreError: 195081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.030110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.540333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6127.190868] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6137.187440] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6145.450415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6145.785925] Lustre: lustre-OST0001: new disk, initializing [ 6145.791444] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6145.798460] Lustre: 196106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6145.916588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6153.874806] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6155.387854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6155.407231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6155.614268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6163.625501] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6169.587394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6169.598225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.611255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6174.930123] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 6174.942916] Lustre: Skipped 4 previous similar messages [ 6180.043809] Lustre: lustre-OST0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 6180.049294] Lustre: Skipped 2 previous similar messages [ 6181.344143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6181.455152] Lustre: server umount lustre-OST0000 complete [ 6185.158877] LustreError: 195081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.764979] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6188.121215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6192.702869] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6192.944738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6212.065787] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.075479] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.081143] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6212.081151] Lustre: Skipped 2 previous similar messages [ 6212.104647] Lustre: Skipped 2 previous similar messages [ 6212.117899] LustreError: Skipped 1 previous similar message [ 6213.698551] Lustre: server umount lustre-OST0001 complete [ 6221.806035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6221.833677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.896765] Lustre: server umount lustre-MDT0000 complete [ 6228.772461] LustreError: 193241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842919 with bad export cookie 13409631262310696025 [ 6228.778662] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.803717] LustreError: 193241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6229.275711] Lustre: server umount lustre-MDT0001 complete [ 6254.679362] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:49:04 (1788842944) [ 6260.827522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.012956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6261.058729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6261.318870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.491439] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6274.322118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.486141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6274.535502] Lustre: 199545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6279.558958] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6288.090996] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6294.734797] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6303.213962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.531615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6311.997719] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6313.103718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6321.084913] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6328.971041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.014833] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6329.266319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6329.274654] Lustre: Skipped 2 previous similar messages [ 6336.621728] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6345.393481] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6348.772583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6348.781464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.788754] Lustre: Skipped 1 previous similar message [ 6348.798240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6348.806961] Lustre: Skipped 4 previous similar messages [ 6354.866650] Lustre: server umount lustre-OST0000 complete [ 6356.463376] LustreError: 201825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.479876] LustreError: 201825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 6376.421306] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.434193] LustreError: Skipped 1 previous similar message [ 6377.589587] Lustre: server umount lustre-OST0001 complete [ 6382.561851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.585126] Lustre: Skipped 3 previous similar messages [ 6382.597750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6382.603067] Lustre: Skipped 7 previous similar messages [ 6388.522141] LustreError: 203618:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000006ef7f197) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6388.608184] Lustre: server umount lustre-MDT0000 complete [ 6392.434508] LustreError: 199540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843083 with bad export cookie 13409631262310697432 [ 6392.437601] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.446814] LustreError: 199540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6392.891147] Lustre: server umount lustre-MDT0001 complete [ 6419.118360] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:51:49 (1788843109) [ 6437.679112] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 6449.814300] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6497.813680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.177419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6498.240426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6498.339172] Lustre: lustre-MDT0000: new disk, initializing [ 6498.449403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.471820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6504.019814] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6517.263872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.469913] Lustre: 207646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6517.609413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6517.615280] Lustre: Skipped 1 previous similar message [ 6517.772424] Lustre: lustre-MDT0001: new disk, initializing [ 6517.974094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6517.994071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6523.256348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6533.930088] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.382946] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6550.207749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.620854] Lustre: lustre-OST0000: new disk, initializing [ 6550.625049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6550.632361] Lustre: 209618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6558.333698] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6560.372330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6560.386284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6560.575274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6569.953679] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6576.236253] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6576.533954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6581.723411] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6581.963703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6586.343196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.360040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.364743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.364752] Lustre: Skipped 2 previous similar messages [ 6586.364789] Lustre: Skipped 2 previous similar messages [ 6586.409787] LustreError: Skipped 1 previous similar message [ 6590.411412] Lustre: server umount lustre-OST0000 complete [ 6600.905261] Lustre: server umount lustre-MDT0000 complete [ 6601.699254] LustreError: 207657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.725460] LustreError: 207657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6605.488634] LustreError: 207639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843296 with bad export cookie 13409631262310698993 [ 6605.497881] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6605.508642] LustreError: 207639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6606.101232] Lustre: server umount lustre-MDT0001 complete [ 6616.744760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.732511] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6633.351149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.375580] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6649.699710] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6657.326101] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6666.947984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6677.056347] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6686.812558] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6692.397229] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6692.653755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6698.534218] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6698.726314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6702.564492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6702.581857] LustreError: Skipped 1 previous similar message [ 6707.585955] Lustre: server umount lustre-OST0000 complete [ 6714.561123] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6721.989014] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6752.324896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.776198] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6767.271174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6767.276989] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6767.277849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6767.293085] Lustre: Skipped 1 previous similar message [ 6771.130425] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6773.414752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6777.829867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.836324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.848116] Lustre: Skipped 5 previous similar messages [ 6777.857870] Lustre: Skipped 7 previous similar messages [ 6782.795433] Lustre: server umount lustre-OST0000 complete [ 6793.715104] Lustre: server umount lustre-MDT0000 complete [ 6798.768520] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843489 with bad export cookie 13409631262310699875 [ 6798.774741] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.776669] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6799.425208] Lustre: server umount lustre-MDT0001 complete [ 6809.646041] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:58:19 (1788843499) [ 6816.992543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.535198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.543625] Lustre: Skipped 6 previous similar messages [ 6822.923137] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6833.976779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.732309] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6849.880455] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.028691] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6865.184391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.290025] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6882.003957] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.248487] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6888.499841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6894.864915] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6895.219146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6902.588592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.854226] Lustre: lustre-OST0001: new disk, initializing [ 6902.863481] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6902.872337] Lustre: 221175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6910.506235] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6911.648888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6911.658579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6911.888055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6922.432866] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6927.935329] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6928.103857] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6934.054837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6934.244691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6937.574122] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6937.588823] LustreError: Skipped 3 previous similar messages [ 6943.067279] Lustre: server umount lustre-OST0001 complete [ 6947.831339] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.847476] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6951.028936] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6951.347787] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6956.350391] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6956.629597] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6988.056542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.909624] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6993.923350] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6993.925305] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6996.879430] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7005.304567] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.754870] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7015.518677] Lustre: server umount lustre-OST0001 complete [ 7034.854266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7034.857100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.862652] Lustre: Skipped 11 previous similar messages [ 7034.887569] Lustre: Skipped 7 previous similar messages [ 7039.537728] Lustre: server umount lustre-OST0000 complete [ 7058.401086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7058.664403] Lustre: server umount lustre-MDT0000 complete [ 7062.898024] LustreError: 217784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843753 with bad export cookie 13409631262310700834 [ 7062.904852] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.915531] LustreError: 217784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7077.344313] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7077.698363] Lustre: server umount lustre-MDT0001 complete [ 7105.896334] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:03:15 (1788843795) [ 7112.914925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.211700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7113.272733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7118.136686] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7126.483409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.601188] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7126.650835] Lustre: 226433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7131.940234] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7140.266695] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7147.247765] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7156.124670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7156.397250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7163.844112] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7172.768679] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7183.094748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7183.228961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7191.346411] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7200.465741] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7202.453194] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7207.607438] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7207.830974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7213.153719] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7213.352339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7220.519910] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff993dd08d0800.ost_server_uuid 50 [ 7222.349232] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff993dd08d0800.ost_server_uuid in IDLE state after 0 sec [ 7223.428652] Lustre: Permanently deactivating lustre-OST0001 [ 7223.444590] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7227.996691] Lustre: Permanently reactivating lustre-OST0001 [ 7228.007922] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7228.012909] Lustre: Skipped 2 previous similar messages [ 7234.022385] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7234.036563] LustreError: Skipped 3 previous similar messages [ 7247.328360] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7247.499738] Lustre: server umount lustre-OST0001 complete [ 7267.296170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7267.600171] Lustre: server umount lustre-OST0000 complete [ 7278.196871] Lustre: server umount lustre-MDT0000 complete [ 7282.895621] LustreError: 226426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843973 with bad export cookie 13409631262310702129 [ 7282.906885] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.910487] LustreError: 226426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7283.281402] Lustre: server umount lustre-MDT0001 complete [ 7310.682991] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:06:40 (1788844000) [ 7319.230495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.528307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7319.557827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7319.563198] Lustre: Skipped 2 previous similar messages [ 7326.583655] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7335.981570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.188390] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7336.216332] Lustre: 233133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7336.492720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7336.501794] Lustre: Skipped 9 previous similar messages [ 7341.305071] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7350.056286] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.999825] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7365.012813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.247729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7365.278614] Lustre: Permanently deactivating lustre-OST0000 [ 7365.290896] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7371.289105] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7372.367457] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7381.161257] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7389.494240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.539669] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7396.332331] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7399.605566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7405.069777] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7407.195114] Lustre: Permanently reactivating lustre-OST0000 [ 7407.216725] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7407.227686] Lustre: Skipped 2 previous similar messages [ 7408.545847] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7408.548985] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000ac3d1e5e, cur 1788844099 deadline 1788844092 last 1788844062 [ 7408.558063] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7408.588907] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7408.600699] Lustre: Skipped 1 previous similar message [ 7408.659100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7427.400279] Lustre: server umount lustre-OST0001 complete [ 7438.609711] Lustre: server umount lustre-OST0000 complete [ 7448.913890] Lustre: server umount lustre-MDT0000 complete [ 7452.992655] LustreError: 235016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844144 with bad export cookie 13409631262310703802 [ 7453.002915] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7453.005806] LustreError: 235016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.839747] Lustre: server umount lustre-MDT0001 complete [ 7466.253559] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7469.051437] Key type lgssc unregistered [ 7469.337857] LNet: 238018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.354659] LNetError: 238018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.374148] LNet: Removed LNI 192.168.202.113@tcp [ 7470.300938] Key type .llcrypt unregistered [ 7470.304050] Key type ._llcrypt unregistered [ 7493.609781] Key type ._llcrypt registered [ 7493.612670] Key type .llcrypt registered [ 7493.727124] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7494.910552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7494.947028] alg: No test for adler32 (adler32-zlib) [ 7495.981387] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7496.224372] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7497.996051] Key type lgssc registered [ 7499.408532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7509.790780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7509.806287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.190688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.867732] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7521.764747] LustreError: 239482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.778919] LustreError: 239482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7523.686232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.816800] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7523.845124] Lustre: 239472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7523.853356] Lustre: Permanently deactivating lustre-MDT0001 [ 7523.864872] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7524.123400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.394419] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7524.401514] LustreError: 239513:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7524.415154] LustreError: 239513:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7524.423454] LustreError: 239513:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7528.941458] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7536.292506] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.595179] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7548.495684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.883134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.926039] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.949144] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7555.042236] LustreError: 241321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.148113] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7556.085571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7561.199655] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7561.229899] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7563.585333] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7566.311690] LustreError: 241720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.321688] LustreError: 241720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7570.218439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7570.428960] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.559399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7575.559757] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7577.002935] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7584.538466] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7588.755117] Lustre: Permanently reactivating lustre-MDT0001 [ 7588.767173] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7588.772121] Lustre: Skipped 2 previous similar messages [ 7593.218324] Lustre: lustre-MDT0001: Client 9d3b3b3b-b025-4f62-bd09-d1f582933228 (at 192.168.202.13@tcp) reconnecting [ 7599.446137] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.523963] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7608.254706] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7608.445300] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7614.111286] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7614.353263] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7617.311264] Lustre: Permanently deactivating lustre-MDT0001 [ 7617.323963] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7617.327685] Lustre: Skipped 2 previous similar messages [ 7626.382170] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7636.963689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.979367] Lustre: Skipped 1 previous similar message [ 7636.980205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7639.455628] Lustre: server umount lustre-MDT0000 complete [ 7642.084266] LustreError: 243040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.095274] LustreError: 243040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7642.952982] LustreError: 239461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844333 with bad export cookie 3006926648987457969 [ 7642.957519] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.969121] LustreError: 239461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7643.242929] Lustre: server umount lustre-MDT0001 complete [ 7649.266890] Lustre: server umount lustre-OST0000 complete [ 7661.646553] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:12:31 (1788844351) [ 7672.233689] Lustre: server umount lustre-OST0001 complete [ 7679.011857] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 7689.534695] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7726.387068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.644852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7726.655623] Lustre: Skipped 2 previous similar messages [ 7726.676250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7726.761658] Lustre: lustre-MDT0000: new disk, initializing [ 7726.858205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.873675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7731.325506] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7742.435663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.558374] Lustre: 247740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7742.619566] LustreError: 247752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.638877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7742.644243] Lustre: Skipped 1 previous similar message [ 7742.706198] Lustre: lustre-MDT0001: new disk, initializing [ 7742.791459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.828050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7742.839988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7747.507625] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7757.918439] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7764.366909] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7771.568086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.915608] Lustre: lustre-OST0000: new disk, initializing [ 7771.921627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7771.931179] Lustre: 249710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7772.034205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.845143] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7781.939053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7781.962518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7782.062092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7787.771118] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7789.782442] LustreError: 247749:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7797.142350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7797.343865] Lustre: lustre-OST0001: new disk, initializing [ 7797.351129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7797.368540] Lustre: 250824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7799.813950] LustreError: 247749:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7799.846502] LustreError: 248823:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7803.040797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7803.051142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7803.205623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7805.703264] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7809.892257] LustreError: 248823:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7818.624881] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7820.824672] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7823.846489] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7823.849529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.857880] LustreError: Skipped 1 previous similar message [ 7823.859815] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7823.859827] Lustre: Skipped 3 previous similar messages [ 7823.888730] Lustre: Skipped 3 previous similar messages [ 7828.964324] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7828.971879] Lustre: Skipped 2 previous similar messages [ 7829.225123] Lustre: server umount lustre-OST0001 complete [ 7834.099594] LustreError: 250202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7834.121715] LustreError: 250202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7848.932330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7848.936183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.945371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.741429] Lustre: server umount lustre-OST0000 complete [ 7859.682572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.683302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7859.692938] Lustre: Skipped 2 previous similar messages [ 7859.707050] Lustre: Skipped 4 previous similar messages [ 7863.322659] Lustre: server umount lustre-MDT0000 complete [ 7866.790681] LustreError: 247732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844557 with bad export cookie 3006926648987460790 [ 7866.794956] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.807539] LustreError: 247732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7881.185717] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7881.740965] Lustre: server umount lustre-MDT0001 complete [ 7894.781125] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7897.935579] Key type lgssc unregistered [ 7898.217691] LNet: 252860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7898.226500] LNetError: 252860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7899.241757] LNet: Removed LNI 192.168.202.113@tcp [ 7900.284433] Key type .llcrypt unregistered [ 7900.286693] Key type ._llcrypt unregistered [ 7944.398158] Key type ._llcrypt registered [ 7944.403079] Key type .llcrypt registered [ 7944.508631] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7946.473698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7946.583132] alg: No test for adler32 (adler32-zlib) [ 7947.842982] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7948.135344] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7949.816238] Key type lgssc registered [ 7951.422492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7961.338468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7961.378501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.752665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7962.787574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7962.925112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.035160] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7980.348530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.445350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7980.481267] Lustre: 255068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7980.822574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.118815] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7995.596598] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8003.094644] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8010.771206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8010.944702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8011.186036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.910222] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8019.137102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8026.960107] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8080.353958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.368241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.385876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8080.873900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.876170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.900293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8085.235818] Lustre: server umount lustre-OST0000 complete [ 8089.872916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8104.571655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.023634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.697583] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8121.390713] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8121.410607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8121.416777] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8121.694504] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8162.278712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.281201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8162.292593] Lustre: Skipped 1 previous similar message [ 8162.303766] Lustre: Skipped 1 previous similar message [ 8167.411136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.424943] Lustre: Skipped 1 previous similar message [ 8168.168743] Lustre: server umount lustre-OST0000 complete [ 8172.513404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.515715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.527309] Lustre: Skipped 1 previous similar message [ 8178.821189] Lustre: server umount lustre-MDT0000 complete [ 8183.355266] LustreError: 255808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844874 with bad export cookie 10496719273008471100 [ 8183.368032] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8183.372673] LustreError: 255808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8183.950702] Lustre: server umount lustre-MDT0001 complete [ 8195.327741] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8198.072315] Key type lgssc unregistered [ 8198.308650] LNet: 260492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8198.311356] LNetError: 260492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8198.323957] LNet: Removed LNI 192.168.202.113@tcp [ 8199.192642] Key type .llcrypt unregistered [ 8199.196579] Key type ._llcrypt unregistered [ 8234.321287] Key type ._llcrypt registered [ 8234.325498] Key type .llcrypt registered [ 8234.539423] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8237.178572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.316683] alg: No test for adler32 (adler32-zlib) [ 8238.610893] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8239.000077] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8240.696331] Key type lgssc registered [ 8242.183978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8252.561974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8252.602806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.080402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.996919] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8264.679060] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8264.718493] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8269.794284] LustreError: 261905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.807085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.254621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8276.844717] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8285.647432] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8293.086105] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8303.106515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.979952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.772284] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8321.728916] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8324.399149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8350.191674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8350.193994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.218452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.219655] Lustre: Skipped 1 previous similar message [ 8350.228397] Lustre: Skipped 1 previous similar message [ 8355.307829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8355.325717] Lustre: Skipped 1 previous similar message [ 8355.651824] Lustre: server umount lustre-OST0000 complete [ 8360.423407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.431751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.539893] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.567149] LustreError: 261906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8365.645783] Lustre: server umount lustre-MDT0000 complete [ 8370.111149] LustreError: 261887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845061 with bad export cookie 8472529530955242996 [ 8370.136347] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8370.153609] LustreError: 261887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8370.687135] Lustre: server umount lustre-MDT0001 complete [ 8387.383854] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8391.367795] Key type lgssc unregistered [ 8391.934049] LNet: 266216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8391.946317] LNetError: 266216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8391.984632] LNet: Removed LNI 192.168.202.113@tcp [ 8393.669564] Key type .llcrypt unregistered [ 8393.674280] Key type ._llcrypt unregistered [ 8416.393850] Key type ._llcrypt registered [ 8416.397923] Key type .llcrypt registered [ 8416.573178] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8417.578476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8417.611020] alg: No test for adler32 (adler32-zlib) [ 8418.768805] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8419.106241] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8420.832109] Key type lgssc registered [ 8422.220115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.319426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8432.366489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.060286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.773133] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8444.393879] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.420673] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8449.506465] LustreError: 267438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.751081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.192663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8455.004573] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8462.936506] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8470.236727] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8478.144739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.558956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.632281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8485.975929] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8493.639734] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.439889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8510.455688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.484174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8511.468782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8511.482247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8511.484548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.250216] Lustre: server umount lustre-OST0000 complete [ 8521.704523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8521.720249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.724137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.749964] Lustre: Skipped 1 previous similar message [ 8521.761049] Lustre: Skipped 2 previous similar messages [ 8523.480623] Lustre: server umount lustre-MDT0000 complete [ 8526.820474] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.836859] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8527.480614] LustreError: 267419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845218 with bad export cookie 7401271006070398763 [ 8527.496311] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.497801] LustreError: 267419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8527.801543] Lustre: server umount lustre-MDT0001 complete [ 8539.598837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8542.261924] Key type lgssc unregistered [ 8542.620914] LNet: 271114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8542.632798] LNetError: 271114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8542.676146] LNet: Removed LNI 192.168.202.113@tcp [ 8543.995918] Key type .llcrypt unregistered [ 8543.997937] Key type ._llcrypt unregistered [ 8576.380973] Key type ._llcrypt registered [ 8576.385383] Key type .llcrypt registered [ 8576.488958] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8578.185182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8578.281085] alg: No test for adler32 (adler32-zlib) [ 8579.653661] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8580.195644] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8582.001945] Key type lgssc registered [ 8582.862241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8592.237372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8592.264365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.862051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.300704] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8603.109262] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.128577] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8608.227436] LustreError: 272530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.973613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.244672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8616.452877] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8626.395884] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8633.960868] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8642.008762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8642.408327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.019980] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8659.170466] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8663.928640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8689.638758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.646342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.666915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8689.669173] Lustre: Skipped 1 previous similar message [ 8689.703595] Lustre: Skipped 1 previous similar message [ 8694.758590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8694.769034] Lustre: Skipped 1 previous similar message [ 8695.821357] Lustre: server umount lustre-OST0000 complete [ 8699.874318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.896513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.911451] Lustre: Skipped 1 previous similar message [ 8704.995341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8705.004333] Lustre: Skipped 1 previous similar message [ 8705.765088] Lustre: server umount lustre-MDT0000 complete [ 8710.127239] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.168061] LustreError: 272531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8711.876138] LustreError: 272512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845402 with bad export cookie 10864055473804001473 [ 8711.882949] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8711.896066] LustreError: 272512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8712.528592] Lustre: server umount lustre-MDT0001 complete [ 8723.949880] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8726.958290] Key type lgssc unregistered [ 8727.325968] LNet: 276837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8727.332366] LNetError: 276837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8727.348228] LNet: Removed LNI 192.168.202.113@tcp [ 8728.459405] Key type .llcrypt unregistered [ 8728.463317] Key type ._llcrypt unregistered [ 8747.355890] Key type ._llcrypt registered [ 8747.359974] Key type .llcrypt registered [ 8747.431934] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8748.718710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8748.793533] alg: No test for adler32 (adler32-zlib) [ 8749.862832] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8750.212958] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8751.970639] Key type lgssc registered [ 8753.833926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8765.782787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8765.817189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.520656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.041273] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8777.703245] LustreError: 278069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.735818] LustreError: 278069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8782.821332] LustreError: 278071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.717693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.148821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8789.296916] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8797.496456] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8803.064693] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8810.453231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.763093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.816559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8817.515615] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8824.988829] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8837.613418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8837.627490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.661301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.635132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.636222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.645660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8842.666383] Lustre: server umount lustre-OST0000 complete [ 8846.818132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8846.825080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.835889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8846.844538] Lustre: Skipped 1 previous similar message [ 8848.867398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.869712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.985575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8853.990027] Lustre: Skipped 2 previous similar messages [ 8861.153609] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8861.355926] Lustre: server umount lustre-MDT0000 complete [ 8863.206759] LustreError: 278051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.270543] LustreError: 278051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8866.239212] LustreError: 278035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845557 with bad export cookie 9929691858305037451 [ 8866.246291] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8866.260328] LustreError: 278035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8866.809452] Lustre: server umount lustre-MDT0001 complete [ 8884.062407] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8888.045237] Key type lgssc unregistered [ 8888.596378] LNet: 281772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8888.618524] LNetError: 281772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8888.681731] LNet: Removed LNI 192.168.202.113@tcp [ 8890.543552] Key type .llcrypt unregistered [ 8890.548915] Key type ._llcrypt unregistered [ 8907.924574] Key type ._llcrypt registered [ 8907.927642] Key type .llcrypt registered [ 8908.029323] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8925.843354] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 8940.941740] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8942.128901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8942.258303] alg: No test for adler32 (adler32-zlib) [ 8943.446595] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8943.730957] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8945.456801] Key type lgssc registered [ 8946.961044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8984.817978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8984.839835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.155565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.217983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8986.324928] Lustre: lustre-MDT0000: new disk, initializing [ 8986.401266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.423292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8991.585707] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9002.932839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.054045] Lustre: 285321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9003.099835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9003.103553] Lustre: Skipped 1 previous similar message [ 9003.199131] Lustre: lustre-MDT0001: new disk, initializing [ 9003.333967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9003.397133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9003.410824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9008.140925] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9018.131369] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9024.063589] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9031.429187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9031.756707] Lustre: lustre-OST0000: new disk, initializing [ 9031.760950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9031.768491] Lustre: 287294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9031.940648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9038.986884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9039.004846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9039.111389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9039.345901] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9051.164454] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9056.648306] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9056.921379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9062.372227] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9062.610371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9064.940271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9064.942477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.987230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9064.993202] Lustre: Skipped 1 previous similar message [ 9070.052637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9070.062132] Lustre: Skipped 1 previous similar message [ 9075.187953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.206478] Lustre: Skipped 1 previous similar message [ 9079.264182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9079.492451] Lustre: server umount lustre-OST0000 complete [ 9085.421220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.427410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.428241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9085.428256] LustreError: Skipped 1 previous similar message [ 9085.442833] Lustre: Skipped 2 previous similar messages [ 9090.491671] Lustre: server umount lustre-MDT0000 complete [ 9094.755591] LustreError: 286119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845785 with bad export cookie 7950033576819325961 [ 9094.756533] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9094.761552] LustreError: 286119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9095.212200] Lustre: server umount lustre-MDT0001 complete [ 9104.845777] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:36:34 (1788845794) [ 9113.596353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.105072] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9114.131862] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9114.199299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9119.206975] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9119.592923] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9124.330521] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.499173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.930389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9132.934471] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9140.676233] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9146.499480] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.578434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.869976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.701994] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9169.064665] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9183.715932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9183.719337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.722715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9183.744602] Lustre: Skipped 1 previous similar message [ 9183.769584] Lustre: Skipped 2 previous similar messages [ 9189.742539] Lustre: server umount lustre-OST0000 complete [ 9193.955358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.971930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.974881] Lustre: Skipped 1 previous similar message [ 9193.993184] Lustre: Skipped 3 previous similar messages [ 9208.288204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9208.761492] Lustre: server umount lustre-MDT0000 complete [ 9209.315025] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.338963] LustreError: 289708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9212.641562] LustreError: 291550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845903 with bad export cookie 7950033576819326843 [ 9212.651416] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9213.033663] Lustre: server umount lustre-MDT0001 complete [ 9224.901960] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9228.185762] Key type lgssc unregistered [ 9228.491176] LNet: 293266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9228.501234] LNetError: 293266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9228.523567] LNet: Removed LNI 192.168.202.113@tcp [ 9229.423640] Key type .llcrypt unregistered [ 9229.425558] Key type ._llcrypt unregistered [ 9256.093719] Key type ._llcrypt registered [ 9256.097162] Key type .llcrypt registered [ 9256.177650] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 9269.074182] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9270.069239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9270.231703] alg: No test for adler32 (adler32-zlib) [ 9271.311311] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9271.568226] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 9273.248339] Key type lgssc registered [ 9274.248744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9326.365418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9326.438637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.802535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.843657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9327.981987] Lustre: lustre-MDT0000: new disk, initializing [ 9328.119475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9328.137750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9332.955233] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9344.918278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9345.059793] Lustre: 297068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9345.129033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9345.131985] Lustre: Skipped 1 previous similar message [ 9345.222435] Lustre: lustre-MDT0001: new disk, initializing [ 9345.379367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9345.435552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9345.450320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9350.118961] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9360.350841] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.223724] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9374.828885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.092559] Lustre: lustre-OST2710: new disk, initializing [ 9375.099720] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9375.107826] Lustre: 299037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9375.262931] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9381.489265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9381.498280] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9381.648883] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9382.754770] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9393.118975] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.803722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9402.092689] Lustre: lustre-OST03e8: new disk, initializing [ 9402.103181] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9402.122425] Lustre: 300101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9402.319140] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9409.123677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9409.132447] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9409.267069] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9409.702832] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9418.515255] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9427.953618] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9428.149234] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9433.088380] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9433.291260] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9439.052586] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9439.299920] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9445.908515] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9446.192650] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9453.031737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9453.038444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.078225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.598400] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.598429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.624978] Lustre: Skipped 1 previous similar message [ 9455.638784] Lustre: Skipped 3 previous similar messages [ 9458.664289] Lustre: server umount lustre-MDT0000 complete [ 9462.232300] LustreError: 297060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846153 with bad export cookie 16876219256545587506 [ 9462.236114] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9462.245307] LustreError: 297060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9462.575461] Lustre: server umount lustre-MDT0001 complete [ 9476.977422] Lustre: server umount lustre-OST2710 complete [ 9490.807790] Lustre: server umount lustre-OST03e8 complete [ 9511.179163] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 9521.942831] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9556.482944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.710828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9556.739348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9556.820815] Lustre: lustre-MDT0000: new disk, initializing [ 9556.947154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.963632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9561.682683] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9573.521520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.643823] Lustre: 305352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9573.695895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9573.699679] Lustre: Skipped 1 previous similar message [ 9573.791575] Lustre: lustre-MDT0001: new disk, initializing [ 9573.904659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9573.989052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9574.003968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9579.004202] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9587.583821] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9595.595657] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9604.031078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9604.425406] Lustre: lustre-OST0000: new disk, initializing [ 9604.429396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9604.435698] Lustre: 307323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9604.551294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.861052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9609.876119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9609.974274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9612.522473] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9624.258941] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9629.703803] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9629.951652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9634.495784] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9634.777763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9640.931892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9640.939750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.942338] LustreError: Skipped 1 previous similar message [ 9640.943109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9640.973779] Lustre: Skipped 2 previous similar messages [ 9646.051463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9646.068825] Lustre: Skipped 2 previous similar messages [ 9651.168155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9651.173508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.191143] Lustre: Skipped 1 previous similar message [ 9651.428966] Lustre: server umount lustre-OST0000 complete [ 9655.778967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9655.786989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.409621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9661.431315] Lustre: Skipped 4 previous similar messages [ 9661.801949] Lustre: server umount lustre-MDT0000 complete [ 9666.537274] LustreError: 305365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.559633] LustreError: 305365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9666.707270] LustreError: 305346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846357 with bad export cookie 16876219256545589200 [ 9666.711106] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.720717] LustreError: 305346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9667.042852] Lustre: server umount lustre-MDT0001 complete [ 9677.921409] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:46:06 (1788846366) [ 9679.909772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9682.194145] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:46:12 (1788846372) [ 9686.730924] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9712.890045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9713.062699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9713.080643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9713.210735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.206247] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9727.913257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.984306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9728.013284] Lustre: 311182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9734.433933] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9761.011122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.228308] Lustre: Denying initial registration attempt from nid 192.168.202.113@tcp, specified as failover [ 9761.237680] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9761.244675] LustreError: 312637:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9761.258156] LustreError: 312637:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9761.267133] LustreError: 312637:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9761.375655] Lustre: server umount lustre-OST0000 complete [ 9761.381445] LustreError: 312637:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9769.449040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9769.465720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9769.486339] Lustre: Skipped 1 previous similar message [ 9769.503935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9782.240635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9782.344958] Lustre: server umount lustre-MDT0000 complete [ 9785.829650] LustreError: 311196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.846755] LustreError: 311196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9787.029771] LustreError: 311177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846478 with bad export cookie 16876219256545590082 [ 9787.040049] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9787.048356] LustreError: 311177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9787.618633] Lustre: server umount lustre-MDT0001 complete [ 9800.916150] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9804.280349] Key type lgssc unregistered [ 9804.518728] LNet: 313758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9804.525643] LNetError: 313758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9804.536315] LNet: Removed LNI 192.168.202.113@tcp [ 9805.282382] Key type .llcrypt unregistered [ 9805.285519] Key type ._llcrypt unregistered [ 9819.069957] Key type ._llcrypt registered [ 9819.072497] Key type .llcrypt registered [ 9819.191563] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9820.781105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9820.836545] alg: No test for adler32 (adler32-zlib) [ 9822.013863] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9822.397724] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 9824.208245] Key type lgssc registered [ 9825.588074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9878.846699] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9889.009939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9889.049810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.312565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9890.342747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9890.550673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9895.575911] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9906.756108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.897662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9906.921288] Lustre: 316540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9907.239110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.317491] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9921.876707] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9928.685343] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9949.103900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9949.396167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9949.655284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.781687] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9963.918561] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9970.153492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9970.155643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9970.192212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9973.191675] Lustre: server umount lustre-OST0000 complete [ 9978.851726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9978.859988] LustreError: Skipped 1 previous similar message [ 9978.864628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9978.889208] Lustre: Skipped 1 previous similar message [ 9978.898166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9978.906037] Lustre: Skipped 1 previous similar message [ 9980.385357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9980.390548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.510692] Lustre: server umount lustre-MDT0000 complete [ 9985.508661] LustreError: 319162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.550841] LustreError: 319162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9988.697959] LustreError: 318407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846679 with bad export cookie 3545981221946997043 [ 9988.698852] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9988.705441] LustreError: 318407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9989.072780] Lustre: server umount lustre-MDT0001 complete [10001.562312] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10004.986575] Key type lgssc unregistered [10005.444810] LNet: 320115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10005.451097] LNetError: 320115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10005.474631] LNet: Removed LNI 192.168.202.113@tcp [10006.444278] Key type .llcrypt unregistered [10006.447231] Key type ._llcrypt unregistered [10033.041687] Key type ._llcrypt registered [10033.048123] Key type .llcrypt registered [10033.176911] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10034.241735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10034.314276] alg: No test for adler32 (adler32-zlib) [10035.706541] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10035.932985] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10037.592811] Key type lgssc registered [10039.097975] Lustre: Echo OBD driver; http://www.lustre.org/ [10048.145655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10048.178189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10049.661988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10054.318847] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10059.248866] LustreError: 321527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.268471] LustreError: 321527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10063.477219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.003228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.799897] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10076.160926] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10082.949341] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10093.172949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.774538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10101.357158] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10110.157961] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10120.163576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10120.172756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.217599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10123.487221] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [10123.751872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.342374] Lustre: server umount lustre-MDT0000 complete [10128.570042] LustreError: 322931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.214130] LustreError: 321508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846822 with bad export cookie 4969870261470873433 [10131.214942] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10131.232288] LustreError: 321508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10133.688711] LustreError: 321523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.711498] LustreError: 321523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10133.739164] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.13@tcp (stopping) [10133.750483] Lustre: Skipped 3 previous similar messages [10133.991204] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.009630] Lustre: Skipped 1 previous similar message [10136.993188] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.995554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10137.024341] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10137.051269] Lustre: Skipped 1 previous similar message [10137.811453] Lustre: server umount lustre-MDT0001 complete [10147.523816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10157.862241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.217539] LustreError: 323360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10176.305104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10176.372812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10182.070313] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10186.730107] LustreError: 323361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.740800] LustreError: 323361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10191.603058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.841066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10191.866241] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10191.992188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.494772] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10198.528743] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10207.804177] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10209.890855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10216.491235] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10218.589836] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10227.681465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10227.686239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.687642] LustreError: Skipped 1 previous similar message [10227.692901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.692909] Lustre: Skipped 1 previous similar message [10227.724140] Lustre: Skipped 1 previous similar message [10231.075868] Lustre: server umount lustre-OST0000 complete [10237.928473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10237.944050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.675203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10248.688352] Lustre: Skipped 9 previous similar messages [10250.208139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10250.405425] Lustre: server umount lustre-MDT0000 complete [10253.797127] LustreError: 325127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.959303] LustreError: 321510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846945 with bad export cookie 4969870261470880580 [10254.961033] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10254.969217] LustreError: 321510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10255.588648] Lustre: server umount lustre-MDT0001 complete [10271.078770] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10274.826927] Key type lgssc unregistered [10275.153281] LNet: 327802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10275.173852] LNetError: 327802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10275.207749] LNet: Removed LNI 192.168.202.113@tcp [10276.198672] Key type .llcrypt unregistered [10276.200417] Key type ._llcrypt unregistered [10307.095588] Key type ._llcrypt registered [10307.104122] Key type .llcrypt registered [10307.280160] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10308.528851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10308.581699] alg: No test for adler32 (adler32-zlib) [10309.882182] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10310.330542] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10312.064231] Key type lgssc registered [10313.898138] Lustre: Echo OBD driver; http://www.lustre.org/ [10323.308199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10323.343863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.988396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.545781] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10334.184151] LustreError: 329218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10334.202781] LustreError: 329218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10339.308622] LustreError: 329219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.679227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.152317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10346.186230] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10355.980786] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10362.335752] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10366.755217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10366.774084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10366.779533] Lustre: Skipped 1 previous similar message [10372.067937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10372.088151] Lustre: Skipped 1 previous similar message [10377.186402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.201393] Lustre: Skipped 1 previous similar message [10380.257828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10380.784634] Lustre: server umount lustre-MDT0000 complete [10382.307257] LustreError: 330214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.322126] LustreError: 330214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10385.593743] LustreError: 329198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847076 with bad export cookie 7937716530166306003 [10385.613714] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.620424] LustreError: 329198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10386.111456] Lustre: server umount lustre-MDT0001 complete [10403.313921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.725979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10403.772254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10403.933237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10409.505844] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10418.936512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.037722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10419.079937] Lustre: 331805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10419.515884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10424.430899] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10434.524879] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.308966] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.697425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.885629] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10451.901512] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10451.908816] LustreError: 333691:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10451.914444] LustreError: 333691:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10451.918797] LustreError: 333691:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10452.022599] Lustre: server umount lustre-OST0000 complete [10452.031395] LustreError: 333691:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10463.542770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.737304] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10463.938805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10471.094751] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10471.607344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10481.092952] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10489.408930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.517556] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10489.606964] Lustre: lustre-OST0001: new disk, initializing [10489.610557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10489.615591] Lustre: 335234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10489.695339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10495.613491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10495.626454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10495.785687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10496.991929] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10506.668638] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10511.338959] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10511.342905] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10511.350626] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10511.363731] Lustre: Skipped 2 previous similar messages [10511.377316] Lustre: Skipped 1 previous similar message [10516.465569] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10516.468730] Lustre: Skipped 1 previous similar message [10524.128379] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10524.298855] Lustre: server umount lustre-OST0001 complete [10526.703754] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.730122] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10531.835227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.843275] LustreError: 334158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.843451] Lustre: Skipped 1 previous similar message [10531.846386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10531.871616] LustreError: 334158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10531.905597] Lustre: Skipped 3 previous similar messages [10534.607626] Lustre: server umount lustre-OST0000 complete [10542.049825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.904040] Lustre: server umount lustre-MDT0000 complete [10547.182240] LustreError: 332050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.201548] LustreError: 332050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10549.201851] LustreError: 331797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847240 with bad export cookie 7937716530166306556 [10549.211428] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10549.218371] LustreError: 331797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10549.630496] Lustre: server umount lustre-MDT0001 complete [10560.393835] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10563.295890] Key type lgssc unregistered [10563.620069] LNet: 337193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10563.640844] LNetError: 337193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10563.654824] LNet: Removed LNI 192.168.202.113@tcp [10564.607297] Key type .llcrypt unregistered [10564.609301] Key type ._llcrypt unregistered [10645.131172] Key type ._llcrypt registered [10645.137080] Key type .llcrypt registered [10645.286255] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [10663.668941] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10665.090328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.134202] alg: No test for adler32 (adler32-zlib) [10666.194741] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10666.405820] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10668.080191] Key type lgssc registered [10669.370320] Lustre: Echo OBD driver; http://www.lustre.org/ [10706.229645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10706.334193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.709810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10707.735586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10707.888913] Lustre: lustre-MDT0000: new disk, initializing [10708.040064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10708.056919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10714.165677] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10727.553350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.734180] Lustre: 342138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10727.788455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10727.792830] Lustre: Skipped 1 previous similar message [10727.857206] Lustre: lustre-MDT0001: new disk, initializing [10727.952571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10727.992976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10728.019725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10733.148774] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10745.447133] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10753.472693] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10761.028251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10761.415432] Lustre: lustre-OST0000: new disk, initializing [10761.419632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10761.442783] Lustre: 344107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10761.594331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10767.605835] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10771.575564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10771.589926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10771.807880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10777.772899] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10782.784314] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10782.925431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.282125] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10788.488163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10792.423104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.425754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10792.448191] Lustre: Skipped 1 previous similar message [10792.461163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10796.860490] Lustre: server umount lustre-OST0000 complete [10802.658071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.661136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.677935] Lustre: Skipped 1 previous similar message [10802.697430] Lustre: Skipped 2 previous similar messages [10807.621714] Lustre: server umount lustre-MDT0000 complete [10811.779896] LustreError: 344106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847502 with bad export cookie 13829379740353323402 [10811.793954] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10811.798623] LustreError: 344106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10812.455986] Lustre: server umount lustre-MDT0001 complete [10821.980096] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:05:11 (1788847511) [10830.390834] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:05:20 (1788847520) [10839.250683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.676063] LustreError: 346863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.695051] LustreError: 346863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10839.765447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.993647] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10845.155417] LustreError: 346864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.251529] LustreError: 346863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.742042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.094364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10857.411456] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10867.130615] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10875.288423] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10884.887858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10885.303195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10894.022589] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10903.589777] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10916.881852] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10918.685578] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10920.500207] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10922.297043] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10924.177763] Lustre: Failing over lustre-MDT0000 [10924.629980] Lustre: server umount lustre-MDT0000 complete [10925.028777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10925.033794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10925.048486] LustreError: 347592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.659747] LustreError: 346864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.697583] LustreError: 346864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10940.903332] LustreError: 347592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.948032] LustreError: 347592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10941.986275] Lustre: 340359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847616/real 1788847616] req@0000000094cc69b0 x1875742406288000/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788847632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10942.027880] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10944.923738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.233254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10953.308631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10956.472607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10958.313684] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10958.338695] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10958.424422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10958.492255] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10970.081894] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10972.231287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10976.443828] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10979.035323] Lustre: Failing over lustre-MDT0000 [10979.401653] Lustre: server umount lustre-MDT0000 complete [10982.154517] LustreError: 346859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10982.173495] LustreError: 346859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10983.907524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.948459] Lustre: Skipped 4 previous similar messages [10999.272614] Lustre: 340359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847674/real 1788847674] req@000000007bd5ba09 x1875742406311040/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788847690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10999.316316] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.380603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.506223] LustreError: 340358:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b38819db x1875742406316032/t0(0) o250->MGC192.168.202.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11009.892262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11009.969285] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11013.820321] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11014.936315] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11015.160635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11015.170200] Lustre: Skipped 2 previous similar messages [11015.219709] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11015.269512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11024.357062] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.899229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11035.627085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.635699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11035.637690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11035.643629] Lustre: Skipped 1 previous similar message [11035.666853] Lustre: Skipped 2 previous similar messages [11037.657835] Lustre: server umount lustre-MDT0000 complete [11040.740319] LustreError: 349355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.758448] LustreError: 349355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [11041.455195] LustreError: 346843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847732 with bad export cookie 13829379740353326342 [11041.464516] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11041.819943] Lustre: server umount lustre-MDT0001 complete [11056.655532] Lustre: server umount lustre-OST0000 complete [11077.106548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.859433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11082.817246] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11092.040754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11096.834722] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11106.314410] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.869673] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11126.076108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11135.969954] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11143.610438] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11147.272661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11157.473560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11157.488257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11157.499408] Lustre: Skipped 1 previous similar message [11157.517159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11157.522516] Lustre: Skipped 1 previous similar message [11161.307386] Lustre: server umount lustre-OST0000 complete [11167.721414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.730791] Lustre: Skipped 1 previous similar message [11171.166755] Lustre: server umount lustre-MDT0000 complete [11172.844738] LustreError: 353233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.863993] LustreError: 353233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11174.930405] LustreError: 353214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847865 with bad export cookie 13829379740353327084 [11174.937428] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11174.946803] LustreError: 353214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11175.388584] Lustre: server umount lustre-MDT0001 complete [11188.357707] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [11191.264988] Key type lgssc unregistered [11191.572887] LNet: 356767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11191.588539] LNetError: 356767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11191.617303] LNet: Removed LNI 192.168.202.113@tcp [11192.584934] Key type .llcrypt unregistered [11192.586613] Key type ._llcrypt unregistered [11219.743236] Key type ._llcrypt registered [11219.745452] Key type .llcrypt registered [11219.841918] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [11220.690979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.795106] alg: No test for adler32 (adler32-zlib) [11221.923665] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [11222.236587] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [11223.976186] Key type lgssc registered [11225.004788] Lustre: Echo OBD driver; http://www.lustre.org/ [11232.701577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11232.717690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.233153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11239.113479] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11244.522333] LustreError: 358280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.548822] LustreError: 358280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11248.776667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.237364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11254.195492] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11261.237804] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11266.181851] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11272.124398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11272.544572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11279.338137] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11279.751623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [11287.628362] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11299.718936] Lustre: server umount lustre-MDT0000 complete [11300.326534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.327515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11300.349161] LustreError: 358277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.888994] LustreError: 360124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847994 with bad export cookie 14321382594225617161 [11303.892571] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.895978] LustreError: 360124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11304.347582] Lustre: server umount lustre-MDT0001 complete [11319.201718] Lustre: server umount lustre-OST0000 complete [11333.689557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.014087] LustreError: 361991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.027601] LustreError: 361991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11334.086346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11338.206354] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11339.239457] LustreError: 361992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.353264] LustreError: 361991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.044504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.385255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11351.485500] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11358.953485] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11365.806866] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11373.128651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11373.412941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11379.570098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [11380.216296] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11386.641520] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.754317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11403.779599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.808358] Lustre: Skipped 2 previous similar messages [11403.811532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11406.754915] Lustre: server umount lustre-MDT0000 complete [11409.382324] LustreError: 361991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.429694] LustreError: 361991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11411.516450] LustreError: 361973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848102 with bad export cookie 14321382594225618267 [11411.530125] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11411.538210] LustreError: 361973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11411.871474] Lustre: server umount lustre-MDT0001 complete [11428.193890] Lustre: server umount lustre-OST0000 complete [11445.017928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.359264] LustreError: 365900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11445.487743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11449.980856] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11459.207207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.181618] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11475.078204] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11481.154530] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11488.000904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11488.528389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.530766] Lustre: Skipped 1 previous similar message [11491.632174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [11496.741366] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11505.724971] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11512.292626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11512.303553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.327229] Lustre: Skipped 2 previous similar messages [11512.335188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11512.342733] Lustre: Skipped 3 previous similar messages [11515.364374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11515.371974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11515.402055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.483956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.490888] Lustre: Skipped 1 previous similar message [11525.609226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11525.614601] Lustre: Skipped 2 previous similar messages [11526.112475] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11526.280744] Lustre: server umount lustre-OST0000 complete [11530.724559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.733743] Lustre: Skipped 1 previous similar message [11535.845540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11535.853665] Lustre: Skipped 3 previous similar messages [11536.400752] Lustre: server umount lustre-MDT0000 complete [11540.155169] LustreError: 367744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848231 with bad export cookie 14321382594225619289 [11540.158797] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.161048] LustreError: 367744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11540.459099] Lustre: server umount lustre-MDT0001 complete [11552.205531] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [11554.913809] Key type lgssc unregistered [11555.310385] LNet: 369436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11555.318931] LNetError: 369436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11555.339324] LNet: Removed LNI 192.168.202.113@tcp [11556.258945] Key type .llcrypt unregistered [11556.261595] Key type ._llcrypt unregistered [11583.380245] Key type ._llcrypt registered [11583.384760] Key type .llcrypt registered [11583.490319] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [11584.596981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11584.736841] alg: No test for adler32 (adler32-zlib) [11586.047415] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [11586.316529] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [11587.953750] Key type lgssc registered [11588.863452] Lustre: Echo OBD driver; http://www.lustre.org/ [11596.038955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11596.059639] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.067959] LustreError: 370875:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11596.087168] LustreError: 370875:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11596.097342] LustreError: 370875:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11596.114101] LustreError: 370875:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11596.119547] LustreError: 370875:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11603.999782] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11604.004937] LustreError: 371273:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11604.045400] LustreError: 371273:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11604.054616] LustreError: 371273:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11604.066351] LustreError: 371273:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11604.076481] LustreError: 371273:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11625.346733] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [11628.109250] Key type lgssc unregistered [11628.545549] LNet: 372276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11628.548769] LNetError: 372276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11628.560579] LNet: Removed LNI 192.168.202.113@tcp [11629.356629] Key type .llcrypt unregistered [11629.358645] Key type ._llcrypt unregistered [11648.710306] Key type ._llcrypt registered [11648.713680] Key type .llcrypt registered [11648.830471] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [11663.667552] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [11664.927558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11665.247207] alg: No test for adler32 (adler32-zlib) [11666.452869] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [11666.705772] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [11668.344651] Key type lgssc registered [11669.333403] Lustre: Echo OBD driver; http://www.lustre.org/ [11704.780112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11704.829073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.089157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11706.144530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11706.279741] Lustre: lustre-MDT0000: new disk, initializing [11706.384207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11706.421701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11711.572766] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11722.223385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.352229] Lustre: 375428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11722.417589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11722.437607] Lustre: Skipped 1 previous similar message [11722.508740] Lustre: lustre-MDT0001: new disk, initializing [11722.580383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11722.607471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11722.613675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11727.235619] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11737.736619] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11744.471537] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11752.143765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.493891] Lustre: lustre-OST0000: new disk, initializing [11752.498732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11752.518332] Lustre: 377397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11752.673446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.667477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11758.677510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11758.882231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11759.424444] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11769.176236] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.459812] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11774.837138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11780.165793] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11780.357404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11784.680873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.683622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11784.708136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.721988] Lustre: Skipped 1 previous similar message [11788.511297] Lustre: server umount lustre-OST0000 complete [11794.401931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11794.408201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.438054] Lustre: Skipped 1 previous similar message [11794.449393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11797.738040] Lustre: server umount lustre-MDT0000 complete [11801.095524] LustreError: 375420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848492 with bad export cookie 8085579470490059756 [11801.099649] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11801.107854] LustreError: 375420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11801.478921] Lustre: server umount lustre-MDT0001 complete [11811.052507] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11813.819709] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:21:43 (1788848503) [11821.077733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.407814] LustreError: 379858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.432409] LustreError: 379858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11821.505958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11826.662198] LustreError: 379859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.840248] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11831.777749] LustreError: 379858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.079041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.374974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11839.742659] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11847.359648] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11855.860270] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11866.865167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11867.680256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11875.594991] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11884.245949] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11892.483293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11892.646295] Lustre: lustre-OST0001: new disk, initializing [11892.651556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11892.659800] Lustre: 382697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11892.788609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11899.382629] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [11902.663349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11902.674617] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11902.795520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11909.891877] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11918.050508] Lustre: lustre-OST0001: Not available for connect from 192.168.202.13@tcp (stopping) [11918.060593] Lustre: Skipped 1 previous similar message [11918.306277] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11918.312720] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11918.330743] LustreError: Skipped 1 previous similar message [11918.375061] Lustre: Skipped 2 previous similar messages [11921.422593] Lustre: server umount lustre-OST0001 complete [11923.133093] LustreError: 383173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.576152] LustreError: 381680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.600081] LustreError: 381680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11938.794267] LustreError: 381678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.824709] LustreError: 381678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11942.374409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11942.376550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11942.390047] LustreError: Skipped 1 previous similar message [11942.404609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11942.416532] Lustre: Skipped 3 previous similar messages [11945.940081] Lustre: server umount lustre-OST0000 complete [11954.155444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11954.162308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11954.169666] Lustre: Skipped 1 previous similar message [11954.170458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11954.170465] Lustre: Skipped 2 previous similar messages [11958.468606] Lustre: server umount lustre-MDT0000 complete [11959.265896] LustreError: 380586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.288218] LustreError: 380586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11962.235984] LustreError: 379840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848653 with bad export cookie 8085579470490060631 [11962.242933] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11962.248362] LustreError: 379840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11968.639458] Lustre: server umount lustre-MDT0001 complete [11981.862891] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [11984.685662] Key type lgssc unregistered [11985.004765] LNet: 384681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.013487] LNetError: 384681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11985.033907] LNet: Removed LNI 192.168.202.113@tcp [11985.868975] Key type .llcrypt unregistered [11985.870941] Key type ._llcrypt unregistered [12042.684778] Key type ._llcrypt registered [12042.689302] Key type .llcrypt registered [12042.873432] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [12044.235905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12044.516416] alg: No test for adler32 (adler32-zlib) [12045.749993] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12046.079510] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [12047.792182] Key type lgssc registered [12048.962921] Lustre: Echo OBD driver; http://www.lustre.org/ [12058.055408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12058.090704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12059.451962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12059.490680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12059.737380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12064.021992] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12074.161318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.244170] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12074.284642] Lustre: 387477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12074.519273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12079.340273] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12087.847521] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12095.201718] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12100.065518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12100.075935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12100.091073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12101.601535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.602511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12101.634430] Lustre: Skipped 1 previous similar message [12103.555135] Lustre: server umount lustre-MDT0000 complete [12107.541257] LustreError: 387470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848798 with bad export cookie 10049041667030036904 [12107.544700] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.553813] LustreError: 387470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12107.796279] Lustre: server umount lustre-MDT0001 complete [12118.464756] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:26:48 (1788848808) [12126.644040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12127.244901] LustreError: 389990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.273087] LustreError: 389990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12127.320522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12131.964875] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12132.321579] LustreError: 389991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.445940] LustreError: 389990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.818230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12142.107550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12146.921925] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12155.884749] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12162.397630] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12171.439715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12171.704549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12171.957378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12180.080348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12189.438753] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12197.712316] Lustre: Permanently deactivating lustre-OST0000 [12197.727163] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12205.128823] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12207.477735] LustreError: 392978:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12207.494542] LustreError: 392978:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12207.505993] LustreError: 392978:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12211.086239] LustreError: 393026:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12211.099910] LustreError: 393026:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12219.460614] Lustre: server umount lustre-OST0000 complete [12221.835886] LustreError: 393276:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12221.839718] LustreError: 393276:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12223.972790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12223.980238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.990784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12227.044685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.045880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12232.162771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12232.173255] Lustre: Skipped 2 previous similar messages [12233.319541] Lustre: server umount lustre-MDT0000 complete [12237.092777] LustreError: 389971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848928 with bad export cookie 10049041667030037485 [12237.096323] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12237.100080] LustreError: 389971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12247.664419] Lustre: server umount lustre-MDT0001 complete [12256.143191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12260.733899] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12263.975163] LustreError: 394436:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12270.317303] Lustre: 394776:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.113@tcp [12272.730829] Lustre: server umount MGS complete [12283.933228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12284.402206] LustreError: 395500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.436493] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12284.444585] Lustre: Skipped 1 previous similar message [12284.511500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12289.209599] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12289.506346] LustreError: 395501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12297.983337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.242244] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12302.573275] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12309.338352] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12314.479629] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12316.119304] Lustre: Permanently reactivating lustre-OST0000 [12316.134380] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12316.137559] Lustre: Skipped 2 previous similar messages [12345.824648] LustreError: 397329:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12353.332253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12353.717680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12353.719803] Lustre: Skipped 1 previous similar message [12359.646574] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12366.715403] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12374.204626] Lustre: server umount lustre-OST0000 complete [12380.129521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12380.136237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.161338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.885627] Lustre: server umount lustre-MDT0000 complete [12391.470346] LustreError: 395482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849082 with bad export cookie 10049041667030038535 [12391.474084] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12391.477686] LustreError: 395482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12401.812493] Lustre: server umount lustre-MDT0001 complete [12412.514571] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [12414.638903] Key type lgssc unregistered [12414.939069] LNet: 399961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12414.944182] LNetError: 399961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12415.974996] LNet: Removed LNI 192.168.202.113@tcp [12417.021119] Key type .llcrypt unregistered [12417.023135] Key type ._llcrypt unregistered [12435.043992] Key type ._llcrypt registered [12435.045689] Key type .llcrypt registered [12435.150051] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [12451.311993] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [12453.003962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12453.034405] alg: No test for adler32 (adler32-zlib) [12454.170316] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12454.402636] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [12456.057126] Key type lgssc registered [12457.690087] Lustre: Echo OBD driver; http://www.lustre.org/ [12493.997860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12494.026433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.307486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12495.325969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12495.408730] Lustre: lustre-MDT0000: new disk, initializing [12495.488021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12495.499743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12499.786698] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12510.109839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12510.307133] Lustre: 403116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12510.364128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12510.367778] Lustre: Skipped 1 previous similar message [12510.429737] Lustre: lustre-MDT0001: new disk, initializing [12510.514738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12510.552471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12510.569390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12516.159277] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12527.173970] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12532.846539] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12540.781031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12541.071372] Lustre: lustre-OST0000: new disk, initializing [12541.076992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12541.084460] Lustre: 405083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12541.188202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12547.259214] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12550.209815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12550.216972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12550.322392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12556.990790] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12561.842140] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12562.074884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12567.082641] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12567.317918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12571.110562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12571.113989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.122561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12575.537741] Lustre: server umount lustre-OST0000 complete [12581.348756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12581.360555] Lustre: Skipped 1 previous similar message [12581.368693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12581.373017] Lustre: Skipped 2 previous similar messages [12585.378308] Lustre: server umount lustre-MDT0000 complete [12588.756978] LustreError: 403108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849279 with bad export cookie 3412356523370177283 [12588.758704] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12588.769828] LustreError: 403108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12589.063534] Lustre: server umount lustre-MDT0001 complete [12597.034308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12598.957521] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:34:48 (1788849288) [12605.990076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12606.368754] LustreError: 407543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.395712] LustreError: 407543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12606.476619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.655180] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12611.560290] LustreError: 407544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.674593] LustreError: 407543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.831393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12619.056665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12623.461726] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12630.951325] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12637.046788] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12643.723645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12644.069884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12649.572744] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12656.428281] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12657.919745] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12661.207522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12665.832530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12665.844722] LustreError: Skipped 1 previous similar message [12665.850276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.870525] Lustre: Skipped 1 previous similar message [12665.873751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12666.851842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12671.477346] Lustre: server umount lustre-OST0000 complete [12675.553818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12675.568822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12675.582775] Lustre: Skipped 1 previous similar message [12675.589951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12675.594145] Lustre: Skipped 2 previous similar messages [12682.209651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12682.214951] Lustre: Skipped 3 previous similar messages [12689.376143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12689.564317] Lustre: server umount lustre-MDT0000 complete [12692.315436] LustreError: 409386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849383 with bad export cookie 3412356523370178165 [12692.316174] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12692.324362] LustreError: 409386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12692.624845] Lustre: server umount lustre-MDT0001 complete [12701.714186] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [12704.089963] Key type lgssc unregistered [12704.312141] LNet: 411266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12704.317606] LNetError: 411266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12704.336263] LNet: Removed LNI 192.168.202.113@tcp [12705.196163] Key type .llcrypt unregistered [12705.198154] Key type ._llcrypt unregistered [12725.334678] Key type ._llcrypt registered [12725.336729] Key type .llcrypt registered [12725.437365] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [12725.682560] Key type .llcrypt unregistered [12725.684568] Key type ._llcrypt unregistered [12743.667865] Key type ._llcrypt registered [12743.670313] Key type .llcrypt registered [12743.777978] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [12745.350848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12745.555361] alg: No test for adler32 (adler32-zlib) [12746.685430] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12747.024427] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [12748.712846] Key type lgssc registered [12749.888517] Lustre: Echo OBD driver; http://www.lustre.org/ [12758.283686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12758.313511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.891758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12765.022586] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12769.252814] LustreError: 413468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.281848] LustreError: 413468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12773.524929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12773.817908] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12773.839954] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12773.851535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12778.982631] LustreError: 413468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.995729] LustreError: 413468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12780.533232] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12784.102356] LustreError: 413463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.120657] LustreError: 413463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12784.141862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12787.822989] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12789.247620] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.269756] LustreError: 413465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12794.480359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12794.837428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12799.405890] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12809.697980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12809.705517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12809.718809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12810.070705] Lustre: server umount lustre-OST0000 complete [12814.822852] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12814.835305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12814.851920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12815.785589] Lustre: server umount lustre-MDT0000 complete [12819.447053] LustreError: 415574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849510 with bad export cookie 17731498715436050360 [12819.470838] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12829.796465] Lustre: server umount lustre-MDT0001 complete [12838.363353] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [12840.156853] Key type lgssc unregistered [12840.366744] LNet: 416615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12840.372217] LNetError: 416615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12840.389163] LNet: Removed LNI 192.168.202.113@tcp [12841.009498] Key type .llcrypt unregistered [12841.011132] Key type ._llcrypt unregistered [12860.218885] Key type ._llcrypt registered [12860.222342] Key type .llcrypt registered [12860.326663] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [12861.124784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12861.179432] alg: No test for adler32 (adler32-zlib) [12862.184582] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [12862.389253] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [12864.080223] Key type lgssc registered [12865.020838] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.075397] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.087631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12886.560416] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12891.681053] LustreError: 417980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [12918.368409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12924.673669] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12932.135529] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12938.420388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12939.007292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12939.077700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12943.187202] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12944.371438] LustreError: 418005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12944.392042] LustreError: 418005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12949.476027] LustreError: 418006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12950.622071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12950.917294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12954.784882] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [12961.327810] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12965.846939] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12974.575318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.578412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12974.584306] Lustre: Skipped 1 previous similar message [12974.604078] Lustre: Skipped 1 previous similar message [12978.032443] Lustre: server umount lustre-OST0000 complete [12981.743121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12981.755689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.782744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12984.803826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12984.808197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.877212] Lustre: server umount lustre-MDT0000 complete [12989.921383] LustreError: 419023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12989.939068] LustreError: 419023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12991.666470] LustreError: 417988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849682 with bad export cookie 4109952473135309636 [12991.674616] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12991.884632] Lustre: server umount lustre-MDT0001 complete [13001.502628] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [13003.517590] Key type lgssc unregistered [13003.835048] LNet: 421549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13003.845977] LNetError: 421549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13003.865572] LNet: Removed LNI 192.168.202.113@tcp [13004.702591] Key type .llcrypt unregistered [13004.707434] Key type ._llcrypt unregistered [13027.690795] Key type ._llcrypt registered [13027.695309] Key type .llcrypt registered [13027.788586] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [13028.521217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13028.550050] alg: No test for adler32 (adler32-zlib) [13029.594362] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [13029.863665] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [13031.520189] Key type lgssc registered [13032.626743] Lustre: Echo OBD driver; http://www.lustre.org/ [13040.369838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13040.383422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13041.916511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13046.615975] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13051.368109] LustreError: 422963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.380141] LustreError: 422963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13053.697854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13054.017542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13058.017394] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13064.888211] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13069.973900] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13075.892295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13076.177869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13082.228225] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13086.402411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13089.753810] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13094.882194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13094.896149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13094.927216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13096.658393] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [13096.942684] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13099.679156] Lustre: server umount lustre-MDT0000 complete [13101.458677] LustreError: 422943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.13@tcp arrived at 1788849792 with bad export cookie 8145595337698376070 [13101.472094] LustreError: 422943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13102.049196] LustreError: 422962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.065334] LustreError: 422962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13102.958426] LustreError: 422942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849793 with bad export cookie 8145595337698375174 [13102.968469] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13104.097282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13104.102465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13104.122484] Lustre: Skipped 1 previous similar message [13104.132297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13104.137270] Lustre: Skipped 3 previous similar messages [13107.171118] LustreError: 422963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13107.171566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13107.190869] LustreError: 422963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13109.138425] Lustre: server umount lustre-OST0000 complete [13112.296078] LustreError: 422962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13112.312448] LustreError: 422962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13115.015747] Lustre: server umount lustre-MDT0001 complete [13124.231674] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [13126.583936] Key type lgssc unregistered [13126.910550] LNet: 426644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13126.918265] LNetError: 426644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13126.930868] LNet: Removed LNI 192.168.202.113@tcp [13127.680347] Key type .llcrypt unregistered [13127.682807] Key type ._llcrypt unregistered [13147.712266] Key type ._llcrypt registered [13147.714683] Key type .llcrypt registered [13147.805683] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [13148.928823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13148.953460] alg: No test for adler32 (adler32-zlib) [13150.007184] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [13150.287330] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [13151.962279] Key type lgssc registered [13153.013782] Lustre: Echo OBD driver; http://www.lustre.org/ [13159.818872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13159.849916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13161.234366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13166.177941] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13171.683334] LustreError: 428037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13171.701048] LustreError: 428037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13173.795403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13174.219299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.397322] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13185.324788] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13191.350146] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13197.947647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13198.424066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13204.700988] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13206.701118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13211.135316] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13217.255434] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.271310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13217.274678] Lustre: Skipped 1 previous similar message [13217.285702] Lustre: Skipped 1 previous similar message [13219.044525] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.13@tcp (stopping) [13222.184309] Lustre: server umount lustre-MDT0001 complete [13222.374379] LustreError: 428038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13222.397099] LustreError: 428038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13224.129898] LustreError: 428033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13227.500073] LustreError: 428034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13227.510444] LustreError: 428034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13232.614446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13232.619439] LustreError: 429082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13232.628225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13232.662672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13234.445462] Lustre: server umount lustre-OST0000 complete [13237.440390] Lustre: server umount lustre-MDT0000 complete [13249.226319] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [13251.380938] Key type lgssc unregistered [13251.670591] LNet: 431720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13251.678846] LNetError: 431720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13251.699952] LNet: Removed LNI 192.168.202.113@tcp [13252.440954] Key type .llcrypt unregistered [13252.442569] Key type ._llcrypt unregistered [13274.407334] Key type ._llcrypt registered [13274.409937] Key type .llcrypt registered [13274.514833] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [13288.997896] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [13289.723894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13289.789025] alg: No test for adler32 (adler32-zlib) [13290.877541] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [13291.094473] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [13292.760195] Key type lgssc registered [13293.806262] Lustre: Echo OBD driver; http://www.lustre.org/ [13330.495722] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [13343.842889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13343.861440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13345.088431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13345.135546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13345.215075] Lustre: lustre-MDT0000: new disk, initializing [13345.318124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13345.333106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13349.326923] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13362.905254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13363.156814] Lustre: lustre-OST0000: new disk, initializing [13363.161559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13363.166195] Lustre: Skipped 1 previous similar message [13363.171286] Lustre: 436748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13363.250362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13368.756504] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13369.377239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13369.391256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13369.432568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13377.432991] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13390.862055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13390.979193] Lustre: 435730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13391.018573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13391.068478] Lustre: lustre-MDT0001: new disk, initializing [13391.165659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13391.203271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13391.221404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13395.435794] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13403.964670] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13404.172628] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13408.314977] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13408.553653] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13414.094730] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.13@tcp (stopping) [13415.920703] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13415.932253] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13415.955981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13415.965110] Lustre: Skipped 1 previous similar message [13419.283488] Lustre: server umount lustre-MDT0001 complete [13426.148791] LustreError: 435743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13426.158688] LustreError: 435743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13431.267443] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.282949] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13435.875196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13435.881181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13435.893233] Lustre: Skipped 1 previous similar message [13435.897891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13436.395884] LustreError: 435743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13441.507265] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13441.508996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13441.524507] Lustre: Skipped 1 previous similar message [13446.626075] LustreError: 435743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13446.628716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13449.184839] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13449.431748] Lustre: server umount lustre-OST0000 complete [13467.104138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13467.454128] Lustre: server umount lustre-MDT0000 complete [13479.594245] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [13481.699850] Key type lgssc unregistered [13481.992353] LNet: 440492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13482.002665] LNetError: 440492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13482.030378] LNet: Removed LNI 192.168.202.113@tcp [13482.748491] Key type .llcrypt unregistered [13482.752495] Key type ._llcrypt unregistered [13518.599825] Key type ._llcrypt registered [13518.601749] Key type .llcrypt registered [13518.684817] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [13518.894456] Key type .llcrypt unregistered [13518.897626] Key type ._llcrypt unregistered [13533.153904] Key type ._llcrypt registered [13533.155446] Key type .llcrypt registered [13533.245542] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [13533.969334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13533.990994] alg: No test for adler32 (adler32-zlib) [13534.956667] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [13535.132716] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [13536.768162] Key type lgssc registered [13537.557924] Lustre: Echo OBD driver; http://www.lustre.org/ [13556.674192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13556.718791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13557.887335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13557.930797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13557.992351] Lustre: lustre-MDT0000: new disk, initializing [13558.044955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13558.055817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13561.339742] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13569.639776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.725837] Lustre: 443849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13569.788382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13569.796167] Lustre: Skipped 1 previous similar message [13569.844830] Lustre: lustre-MDT0001: new disk, initializing [13569.927751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13569.962015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13569.970788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13573.575589] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13581.943748] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13586.900955] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13591.853094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13592.029948] Lustre: lustre-OST0000: new disk, initializing [13592.031869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13592.034727] Lustre: 445820:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13592.104509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13597.109265] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13600.877933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13600.891147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13600.940084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13605.419823] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13611.491553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13611.503739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13611.508150] Lustre: Skipped 1 previous similar message [13611.524084] Lustre: Skipped 2 previous similar messages [13616.612224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13616.620374] Lustre: Skipped 2 previous similar messages [13617.660649] Lustre: server umount lustre-MDT0000 complete [13620.412295] LustreError: 443840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850311 with bad export cookie 6035825821964144443 [13620.416908] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13620.421321] LustreError: 443840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13620.670852] Lustre: server umount lustre-MDT0001 complete [13633.978304] Lustre: server umount lustre-OST0000 complete [13642.856259] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:52:13 (1788850333) [13649.005947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.412611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13653.328616] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13654.512779] LustreError: 447847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13654.545047] LustreError: 447847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13659.622266] LustreError: 447846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13660.606560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13660.840130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13663.793869] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13669.718340] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13674.202402] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13692.348474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13692.520078] Lustre: Found index 0 for lustre-OST0000, updating log [13692.716510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13694.766857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13697.688573] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [13703.195244] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13707.042705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13707.043372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13707.064114] Lustre: Skipped 1 previous similar message [13709.158716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13709.170103] Lustre: Skipped 2 previous similar messages [13712.242112] Lustre: server umount lustre-OST0000 complete [13726.581885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13726.591397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13726.592425] Lustre: Skipped 1 previous similar message [13726.609335] Lustre: Skipped 3 previous similar messages [13732.890580] Lustre: server umount lustre-MDT0000 complete [13735.616836] LustreError: 447828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850426 with bad export cookie 6035825821964145836 [13735.626460] LustreError: 447828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13735.628970] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13735.891717] Lustre: server umount lustre-MDT0001 complete [13742.627515] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:53:53 (1788850433) [13760.416462] LNet: 1 local NIs in recovery (showing 1): 192.168.202.113@tcp [14360.498164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14360.891137] LustreError: 454266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14360.997465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14366.178677] LustreError: 454267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14366.653828] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14370.291362] LustreError: 454266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14376.430751] LustreError: 454266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14376.451353] LustreError: 454266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14377.160091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14377.472879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14382.410521] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14390.541689] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14397.484107] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14412.769715] LNet: 1 local NIs in recovery (showing 1): 192.168.202.113@tcp [14417.976855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14418.276820] Lustre: Found index 0 for lustre-OST0000, updating log [14418.623542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14425.019062] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14433.898444] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14436.321840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14436.326425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14436.344094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14436.346854] Lustre: Skipped 2 previous similar messages [14442.645560] Lustre: server umount lustre-OST0000 complete [14448.675818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14448.691584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14448.697606] Lustre: Skipped 1 previous similar message [14448.719733] Lustre: Skipped 1 previous similar message [14450.797904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14450.817307] Lustre: Skipped 3 previous similar messages [14454.313812] Lustre: server umount lustre-MDT0000 complete [14454.885054] LustreError: 454262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14454.914727] LustreError: 454262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14459.308552] LustreError: 454246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851150 with bad export cookie 6035825821964146907 [14459.318847] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14459.350050] LustreError: 454246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14460.019854] Lustre: server umount lustre-MDT0001 complete [14472.224352] LNet: 1 local NIs in recovery (showing 1): 192.168.202.113@tcp [14492.778512] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [14507.112841] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [14547.252838] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:07:17 (1788851237) [14568.025458] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [14570.211411] Key type lgssc unregistered [14570.446962] LNet: 460893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14570.458033] LNetError: 460893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14682.087439] LNet: Removed LNI 192.168.202.113@tcp [14683.331492] Key type .llcrypt unregistered [14683.338642] Key type ._llcrypt unregistered [14705.249065] Key type ._llcrypt registered [14705.252194] Key type .llcrypt registered [14705.403658] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [14706.608285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14706.729255] alg: No test for adler32 (adler32-zlib) [14708.690966] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [14708.991273] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [14710.729286] Key type lgssc registered [14711.691606] Lustre: Echo OBD driver; http://www.lustre.org/ [14716.547315] LNet: Added LNI 192.168.202.113@tcp100 [8/256/0/180] [14717.509866] LNet: Added LNI 192.168.202.113@tcp101 [8/256/0/180] [14718.355715] LNet: Added LNI 192.168.202.113@tcp102 [8/256/0/180] [14719.297478] LNet: Added LNI 192.168.202.113@tcp103 [8/256/0/180] [14720.214248] LNet: Added LNI 192.168.202.113@tcp104 [8/256/0/180] [14721.079153] LNet: Added LNI 192.168.202.113@tcp105 [8/256/0/180] [14721.967265] LNet: Added LNI 192.168.202.113@tcp106 [8/256/0/180] [14723.077390] LNet: Added LNI 192.168.202.113@tcp107 [8/256/0/180] [14724.019595] LNet: Added LNI 192.168.202.113@tcp108 [8/256/0/180] [14724.821881] LNet: Added LNI 192.168.202.113@tcp109 [8/256/0/180] [14725.666644] LNet: Added LNI 192.168.202.113@tcp110 [8/256/0/180] [14726.610197] LNet: Added LNI 192.168.202.113@tcp111 [8/256/0/180] [14727.572299] LNet: Added LNI 192.168.202.113@tcp112 [8/256/0/180] [14728.470714] LNet: Added LNI 192.168.202.113@tcp113 [8/256/0/180] [14729.391691] LNet: Added LNI 192.168.202.113@tcp114 [8/256/0/180] [14730.325343] LNet: Added LNI 192.168.202.113@tcp115 [8/256/0/180] [14731.044476] LNet: Added LNI 192.168.202.113@tcp116 [8/256/0/180] [14731.869400] LNet: Added LNI 192.168.202.113@tcp117 [8/256/0/180] [14732.800596] LNet: Added LNI 192.168.202.113@tcp118 [8/256/0/180] [14733.473240] LNet: Added LNI 192.168.202.113@tcp119 [8/256/0/180] [14734.280287] LNet: Added LNI 192.168.202.113@tcp120 [8/256/0/180] [14735.298375] LNet: Added LNI 192.168.202.113@tcp121 [8/256/0/180] [14735.897050] LNet: Added LNI 192.168.202.113@tcp122 [8/256/0/180] [14736.412920] LNet: Added LNI 192.168.202.113@tcp123 [8/256/0/180] [14737.208906] LNet: Added LNI 192.168.202.113@tcp124 [8/256/0/180] [14738.514235] LNet: Added LNI 192.168.202.113@tcp125 [8/256/0/180] [14739.634954] LNet: Added LNI 192.168.202.113@tcp126 [8/256/0/180] [14740.648606] LNet: Added LNI 192.168.202.113@tcp127 [8/256/0/180] [14741.565987] LNet: Added LNI 192.168.202.113@tcp128 [8/256/0/180] [14742.440373] LNet: Added LNI 192.168.202.113@tcp129 [8/256/0/180] [14743.163363] LNet: Added LNI 192.168.202.113@tcp130 [8/256/0/180] [14743.941388] LNet: Added LNI 192.168.202.113@tcp131 [8/256/0/180] [14745.305902] LNet: Added LNI 192.168.202.113@tcp132 [8/256/0/180] [14746.222435] LNet: Added LNI 192.168.202.113@tcp133 [8/256/0/180] [14747.366287] LNet: Added LNI 192.168.202.113@tcp134 [8/256/0/180] [14748.230238] LNet: Added LNI 192.168.202.113@tcp135 [8/256/0/180] [14755.356473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14755.384788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14756.565332] Lustre: 463922:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.113@tcp: can't add NID 192.168.202.113@tcp130: rc = -75 [14756.647152] LNetError: 463943:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14756.661224] LNetError: Protocol error connecting to 192.168.202.113@tcp113 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14756.685413] LNetError: Skipped 6 previous similar messages [14756.875496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14756.920180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14757.116321] Lustre: lustre-MDT0000: new disk, initializing [14757.155131] LNet: 5 local NIs in recovery (showing 5): 192.168.202.113@tcp125, 192.168.202.113@tcp126, 192.168.202.113@tcp127, 192.168.202.113@tcp128, 192.168.202.113@tcp124 [14757.157185] LNetError: 463948:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14757.227733] LNetError: 463948:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14757.234865] LNetError: Protocol error connecting to 192.168.202.113@tcp102 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14757.342319] Lustre: 463980:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.113@tcp: can't add NID 192.168.202.113@tcp130: rc = -75 [14757.356473] Lustre: 463980:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14757.386040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14757.413798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14763.321513] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14775.192818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14775.351825] Lustre: 463940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14775.429892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14775.434122] Lustre: Skipped 1 previous similar message [14775.525807] Lustre: lustre-MDT0001: new disk, initializing [14775.569615] Lustre: 464742:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.113@tcp: can't add NID 192.168.202.113@tcp130: rc = -75 [14775.578388] Lustre: 464742:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14775.741596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14775.790895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14775.801781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14780.641584] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14783.549171] Lustre: 465123:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.113@tcp: can't add NID 192.168.202.113@tcp130: rc = -75 [14783.562541] Lustre: 465123:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14787.945952] LNetError: 461380:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14787.967502] LNetError: Protocol error connecting to 192.168.202.113@tcp113 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14790.713390] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14797.153596] LNetError: 463944:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14797.164609] LNetError: Protocol error connecting to 192.168.202.113@tcp102 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14799.139386] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14808.251721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14808.718431] Lustre: lustre-OST0000: new disk, initializing [14808.731559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14808.742018] Lustre: 465915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14808.884338] Lustre: 465920:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.113@tcp: can't add NID 192.168.202.113@tcp130: rc = -75 [14808.892139] Lustre: 465920:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14808.939605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14810.471560] LNetError: 463947:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14810.496431] LNetError: 463947:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14810.511087] LNetError: Protocol error connecting to 192.168.202.113@tcp102 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14810.525585] LNetError: Skipped 2 previous similar messages [14816.432814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14816.451984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14816.667841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14817.669628] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [14823.780467] LNetError: 463942:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14823.799406] LNetError: 463942:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14823.805587] LNetError: Protocol error connecting to 192.168.202.113@tcp102 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14823.815911] LNetError: Skipped 1 previous similar message [14829.182587] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14842.350076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14842.363646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14842.366815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14842.401503] LustreError: Skipped 1 previous similar message [14844.748369] Lustre: server umount lustre-OST0000 complete [14849.377899] LNetError: 463948:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.113 [14849.391209] LNetError: 463948:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14849.398148] LNetError: Protocol error connecting to 192.168.202.113@tcp102 at host 192.168.202.113:988: is it running a compatible version of Lustre? [14849.404716] LNetError: Skipped 1 previous similar message [14852.579558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14852.581735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14852.593971] Lustre: Skipped 1 previous similar message [14852.597228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14852.612512] Lustre: Skipped 2 previous similar messages [14855.794336] Lustre: server umount lustre-MDT0000 complete [14860.084967] LustreError: 464751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851551 with bad export cookie 10866905209031846115 [14860.089251] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14860.096939] LustreError: 464751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14860.437236] Lustre: server umount lustre-MDT0001 complete [14873.783880] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [14877.555801] Key type lgssc unregistered [14878.123528] LNet: 467830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14878.135563] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14878.149443] LNet: Removed LNI 192.168.202.113@tcp [14879.200267] LNet: Removed LNI 192.168.202.113@tcp100 [14879.230633] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14879.250401] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14879.259167] LNet: Removed LNI 192.168.202.113@tcp101 [14880.288330] LNet: Removed LNI 192.168.202.113@tcp102 [14880.299919] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14880.318658] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14880.329416] LNet: Removed LNI 192.168.202.113@tcp103 [14881.376516] LNet: Removed LNI 192.168.202.113@tcp104 [14882.400332] LNet: Removed LNI 192.168.202.113@tcp105 [14882.402545] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14882.432249] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14882.452106] LNet: Removed LNI 192.168.202.113@tcp106 [14883.552398] LNet: Removed LNI 192.168.202.113@tcp107 [14884.640274] LNet: Removed LNI 192.168.202.113@tcp108 [14885.664209] LNet: Removed LNI 192.168.202.113@tcp109 [14886.752345] LNet: Removed LNI 192.168.202.113@tcp110 [14886.759961] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14886.767912] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14886.791950] LNet: Removed LNI 192.168.202.113@tcp111 [14887.840631] LNet: Removed LNI 192.168.202.113@tcp112 [14888.864480] LNet: Removed LNI 192.168.202.113@tcp113 [14889.888198] LNet: Removed LNI 192.168.202.113@tcp114 [14890.913072] LNet: Removed LNI 192.168.202.113@tcp115 [14891.936621] LNet: Removed LNI 192.168.202.113@tcp116 [14892.962080] LNet: Removed LNI 192.168.202.113@tcp117 [14893.986300] LNet: Removed LNI 192.168.202.113@tcp118 [14895.008307] LNet: Removed LNI 192.168.202.113@tcp119 [14895.011220] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14895.018295] LNetError: 467830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14895.024280] LNet: Removed LNI 192.168.202.113@tcp120 [14896.096173] LNet: Removed LNI 192.168.202.113@tcp121 [14897.121031] LNet: Removed LNI 192.168.202.113@tcp122 [14898.146909] LNet: Removed LNI 192.168.202.113@tcp123 [14899.168698] LNet: Removed LNI 192.168.202.113@tcp124 [14900.256118] LNet: Removed LNI 192.168.202.113@tcp125 [14901.280204] LNet: Removed LNI 192.168.202.113@tcp126 [14902.304419] LNet: Removed LNI 192.168.202.113@tcp127 [14903.329154] LNet: Removed LNI 192.168.202.113@tcp128 [14904.352192] LNet: Removed LNI 192.168.202.113@tcp129 [14904.354817] LNet: Removed LNI 192.168.202.113@tcp130 [14904.357199] LNet: Removed LNI 192.168.202.113@tcp131 [14904.359626] LNet: Removed LNI 192.168.202.113@tcp132 [14904.361784] LNet: Removed LNI 192.168.202.113@tcp133 [14904.363923] LNet: Removed LNI 192.168.202.113@tcp134 [14904.375701] LNet: Removed LNI 192.168.202.113@tcp135 [14904.901493] Key type .llcrypt unregistered [14904.911899] Key type ._llcrypt unregistered [14932.868249] Key type ._llcrypt registered [14932.871597] Key type .llcrypt registered [14932.998919] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [14953.707892] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [14954.808074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14955.017811] alg: No test for adler32 (adler32-zlib) [14956.090324] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [14956.510861] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [14958.248192] Key type lgssc registered [14959.461891] Lustre: Echo OBD driver; http://www.lustre.org/ [15009.769319] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:14:59 (1788851699) [15033.029796] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [15036.510697] Key type lgssc unregistered [15036.824136] LNet: 471921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15036.836502] LNetError: 471921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15036.866409] LNet: Removed LNI 192.168.202.113@tcp [15038.092727] Key type .llcrypt unregistered [15038.099795] Key type ._llcrypt unregistered [15059.648808] Key type ._llcrypt registered [15059.654336] Key type .llcrypt registered [15059.830264] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [15061.259148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15061.294105] alg: No test for adler32 (adler32-zlib) [15062.735951] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [15063.278940] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [15065.112239] Key type lgssc registered [15066.606856] Lustre: Echo OBD driver; http://www.lustre.org/ [15077.292170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15077.321970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15078.716765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15078.765835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15078.822916] Lustre: lustre-MDT0000: new disk, initializing [15078.907516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15078.930525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15084.218702] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15095.689645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15095.841770] Lustre: 473140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15095.943229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15095.947754] Lustre: Skipped 1 previous similar message [15096.017917] Lustre: lustre-MDT0001: new disk, initializing [15096.207120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15096.286865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15096.298044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15101.067450] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15111.840843] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15118.093952] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15125.801830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15126.240239] Lustre: lustre-OST0000: new disk, initializing [15126.244805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15126.251791] Lustre: 475111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15126.327755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15134.027020] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15135.835868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15135.845835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15135.984273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15144.899616] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15150.335702] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing lnet_if_list [15156.567566] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15156.585961] LNet: Added LNI 192.168.202.113@tcp10 [8/256/0/180] [15162.586223] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15162.619805] LNet: Added LNI 192.168.202.113@tcp11 [8/256/0/180] [15167.887228] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15167.916601] LNet: Added LNI 192.168.202.113@tcp12 [8/256/0/180] [15172.367699] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15172.409899] LNet: Added LNI 192.168.202.113@tcp13 [8/256/0/180] [15179.501878] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15179.555331] LNet: Added LNI 192.168.202.113@tcp14 [8/256/0/180] [15185.587767] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15185.654081] LNet: Added LNI 192.168.202.113@tcp15 [8/256/0/180] [15193.007920] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15193.105803] LNet: Added LNI 192.168.202.113@tcp16 [8/256/0/180] [15199.280374] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15199.306981] LNet: Added LNI 192.168.202.113@tcp17 [8/256/0/180] [15204.469868] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15204.507627] LNet: Added LNI 192.168.202.113@tcp18 [8/256/0/180] [15209.797872] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15209.818743] LNet: Added LNI 192.168.202.113@tcp19 [8/256/0/180] [15216.038525] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15216.099529] LNet: Added LNI 192.168.202.113@tcp20 [8/256/0/180] [15223.703911] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15223.748058] LNet: Added LNI 192.168.202.113@tcp21 [8/256/0/180] [15230.974452] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15231.016976] LNet: Added LNI 192.168.202.113@tcp22 [8/256/0/180] [15237.517893] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15237.544851] LNet: Added LNI 192.168.202.113@tcp23 [8/256/0/180] [15245.169789] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15245.202799] LNet: Added LNI 192.168.202.113@tcp24 [8/256/0/180] [15252.655847] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15252.693383] LNet: Added LNI 192.168.202.113@tcp25 [8/256/0/180] [15259.641331] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15259.672265] LNet: Added LNI 192.168.202.113@tcp26 [8/256/0/180] [15266.868498] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15266.909539] LNet: Added LNI 192.168.202.113@tcp27 [8/256/0/180] [15274.522732] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15274.574297] LNet: Added LNI 192.168.202.113@tcp28 [8/256/0/180] [15282.922729] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15282.955985] LNet: Added LNI 192.168.202.113@tcp29 [8/256/0/180] [15292.797703] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15292.838522] LNet: Added LNI 192.168.202.113@tcp30 [8/256/0/180] [15301.374855] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15301.410329] LNet: Added LNI 192.168.202.113@tcp31 [8/256/0/180] [15309.348049] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15309.369293] LNet: Added LNI 192.168.202.113@tcp32 [8/256/0/180] [15315.573030] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15315.601855] LNet: Added LNI 192.168.202.113@tcp33 [8/256/0/180] [15322.781818] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15322.815696] LNet: Added LNI 192.168.202.113@tcp34 [8/256/0/180] [15329.685651] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15329.703168] LNet: Added LNI 192.168.202.113@tcp35 [8/256/0/180] [15337.944109] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15337.961889] LNet: Added LNI 192.168.202.113@tcp36 [8/256/0/180] [15347.448490] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15347.519026] LNet: Added LNI 192.168.202.113@tcp37 [8/256/0/180] [15357.534791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15357.569925] LNet: Added LNI 192.168.202.113@tcp38 [8/256/0/180] [15365.814608] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15365.855583] LNet: Added LNI 192.168.202.113@tcp39 [8/256/0/180] [15373.711583] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15373.734635] LNet: Added LNI 192.168.202.113@tcp40 [8/256/0/180] [15380.998028] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15381.020947] LNet: Added LNI 192.168.202.113@tcp41 [8/256/0/180] [15391.559853] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15391.584356] LNet: Added LNI 192.168.202.113@tcp42 [8/256/0/180] [15403.473071] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15403.513667] LNet: Added LNI 192.168.202.113@tcp43 [8/256/0/180] [15421.739815] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15421.791285] LNet: Added LNI 192.168.202.113@tcp44 [8/256/0/180] [15433.148229] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15433.171109] LNet: Added LNI 192.168.202.113@tcp45 [8/256/0/180] [15441.923325] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15441.940661] LNet: Added LNI 192.168.202.113@tcp46 [8/256/0/180] [15449.803844] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15449.871660] LNet: Added LNI 192.168.202.113@tcp47 [8/256/0/180] [15461.459514] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15461.511965] LNet: Added LNI 192.168.202.113@tcp48 [8/256/0/180] [15474.628653] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15474.643193] LNet: Added LNI 192.168.202.113@tcp49 [8/256/0/180] [15484.425796] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15484.465538] LNet: Added LNI 192.168.202.113@tcp50 [8/256/0/180] [15505.889447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15505.894931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15505.914347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15509.987081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15509.993390] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15515.107647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15515.120039] Lustre: Skipped 4 previous similar messages [15520.224142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15520.233250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15520.248898] Lustre: Skipped 2 previous similar messages [15520.590690] Lustre: server umount lustre-MDT0000 complete [15524.931879] LustreError: 473134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852215 with bad export cookie 5450691252103631385 [15524.932835] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15524.956554] LustreError: 473134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15525.348459] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15525.359190] Lustre: Skipped 1 previous similar message [15525.461428] Lustre: server umount lustre-MDT0001 complete [15544.296441] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [15553.560394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15570.647321] LustreError: 475104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15570.687472] LustreError: 475104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15570.781335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15575.610874] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15576.033424] LustreError: 475105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15580.138120] LustreError: 475104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15580.142558] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15585.261305] LustreError: 475105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15586.034746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15586.539509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15589.611391] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15592.717286] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15603.159980] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15608.401462] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15632.356895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15632.372422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15632.391083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15637.781223] Lustre: server umount lustre-OST0000 complete [15642.594623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15642.602098] LustreError: Skipped 1 previous similar message [15642.608247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15642.617569] Lustre: Skipped 1 previous similar message [15642.624363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15642.632859] Lustre: Skipped 3 previous similar messages [15647.942512] Lustre: server umount lustre-MDT0000 complete [15651.305696] LustreError: 499368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15651.321629] LustreError: 499368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15651.986570] LustreError: 475110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852343 with bad export cookie 5450691252103634024 [15651.987705] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15651.995747] LustreError: 475110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15652.371196] Lustre: server umount lustre-MDT0001 complete [15664.541157] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [15667.506684] Key type lgssc unregistered [15668.136337] LNet: 502166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15668.157965] LNetError: 502166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15668.175310] LNet: Removed LNI 192.168.202.113@tcp [15668.183858] LNet: Removed LNI 192.168.202.113@tcp10 [15668.198122] LNet: Removed LNI 192.168.202.113@tcp11 [15668.203884] LNet: Removed LNI 192.168.202.113@tcp12 [15668.211890] LNet: Removed LNI 192.168.202.113@tcp13 [15668.216529] LNet: Removed LNI 192.168.202.113@tcp14 [15668.223549] LNet: Removed LNI 192.168.202.113@tcp15 [15668.227494] LNet: Removed LNI 192.168.202.113@tcp16 [15668.230262] LNet: Removed LNI 192.168.202.113@tcp17 [15668.233551] LNet: Removed LNI 192.168.202.113@tcp18 [15668.235550] LNet: Removed LNI 192.168.202.113@tcp19 [15668.243602] LNet: Removed LNI 192.168.202.113@tcp20 [15668.249520] LNet: Removed LNI 192.168.202.113@tcp21 [15668.253483] LNet: Removed LNI 192.168.202.113@tcp22 [15668.256595] LNet: Removed LNI 192.168.202.113@tcp23 [15668.260683] LNet: Removed LNI 192.168.202.113@tcp24 [15668.266309] LNet: Removed LNI 192.168.202.113@tcp25 [15668.269480] LNet: Removed LNI 192.168.202.113@tcp26 [15668.278612] LNet: Removed LNI 192.168.202.113@tcp27 [15668.282519] LNet: Removed LNI 192.168.202.113@tcp28 [15668.284820] LNet: Removed LNI 192.168.202.113@tcp29 [15668.288905] LNet: Removed LNI 192.168.202.113@tcp30 [15668.294594] LNet: Removed LNI 192.168.202.113@tcp31 [15668.299461] LNet: Removed LNI 192.168.202.113@tcp32 [15668.302950] LNet: Removed LNI 192.168.202.113@tcp33 [15668.306783] LNet: Removed LNI 192.168.202.113@tcp34 [15668.308900] LNet: Removed LNI 192.168.202.113@tcp35 [15668.312282] LNet: Removed LNI 192.168.202.113@tcp36 [15668.315156] LNet: Removed LNI 192.168.202.113@tcp37 [15668.320220] LNet: Removed LNI 192.168.202.113@tcp38 [15668.323195] LNet: Removed LNI 192.168.202.113@tcp39 [15668.326627] LNet: Removed LNI 192.168.202.113@tcp40 [15668.329621] LNet: Removed LNI 192.168.202.113@tcp41 [15668.333835] LNet: Removed LNI 192.168.202.113@tcp42 [15668.339035] LNet: Removed LNI 192.168.202.113@tcp43 [15668.343334] LNet: Removed LNI 192.168.202.113@tcp44 [15668.349551] LNet: Removed LNI 192.168.202.113@tcp45 [15668.353869] LNet: Removed LNI 192.168.202.113@tcp46 [15668.357943] LNet: Removed LNI 192.168.202.113@tcp47 [15668.367356] LNet: Removed LNI 192.168.202.113@tcp48 [15668.379317] LNet: Removed LNI 192.168.202.113@tcp49 [15668.393336] LNet: Removed LNI 192.168.202.113@tcp50 [15669.475117] Key type .llcrypt unregistered [15669.479452] Key type ._llcrypt unregistered [15696.312173] Key type ._llcrypt registered [15696.315567] Key type .llcrypt registered [15696.486240] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [15713.233606] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [15714.461822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15714.650271] alg: No test for adler32 (adler32-zlib) [15715.791492] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [15716.226721] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [15717.928663] Key type lgssc registered [15719.151793] Lustre: Echo OBD driver; http://www.lustre.org/ [15758.813803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15758.839969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15760.232611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15760.321131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15760.488498] Lustre: lustre-MDT0000: new disk, initializing [15760.631923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15760.660471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15765.997346] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15781.777438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15782.004256] Lustre: 505362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15782.087559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15782.089536] Lustre: Skipped 1 previous similar message [15782.163621] Lustre: lustre-MDT0001: new disk, initializing [15782.456896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15782.568436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15782.607744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15789.953796] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15803.764620] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15811.122893] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15820.934853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15821.180894] Lustre: *** cfs_fail_loc=90f, val=0*** [15821.333714] Lustre: lustre-OST0000: new disk, initializing [15821.337718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15821.344345] Lustre: 507381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15821.413445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15828.928864] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15830.129697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15830.141672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15830.317784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15841.444385] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15856.098038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15856.112600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15856.135330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15856.146967] Lustre: Skipped 1 previous similar message [15859.112866] Lustre: server umount lustre-OST0000 complete [15864.290355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15864.293614] LustreError: Skipped 1 previous similar message [15864.299900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15864.308799] Lustre: Skipped 1 previous similar message [15864.313038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15866.337737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15866.340390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15870.296991] Lustre: server umount lustre-MDT0000 complete [15874.604903] LustreError: 505353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852565 with bad export cookie 9284966966976938746 [15874.608955] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15874.631840] LustreError: 505353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15875.597140] Lustre: server umount lustre-MDT0001 complete [15888.262392] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [15891.941321] Key type lgssc unregistered [15892.432829] LNet: 509289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15892.448664] LNetError: 509289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15892.486323] LNet: Removed LNI 192.168.202.113@tcp [15893.716971] Key type .llcrypt unregistered [15893.724481] Key type ._llcrypt unregistered [15914.794113] Key type ._llcrypt registered [15914.795717] Key type .llcrypt registered [15914.888823] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [15916.286907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15916.308036] alg: No test for adler32 (adler32-zlib) [15917.371882] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [15917.634637] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [15919.368788] Key type lgssc registered [15920.268667] Lustre: Echo OBD driver; http://www.lustre.org/ [15927.806485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15927.829140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15929.248257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15934.542455] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15939.554174] LustreError: 510554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15939.571377] LustreError: 510554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15944.325079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15944.578403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15949.617838] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15957.413210] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15964.088658] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15972.437915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15972.752877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15980.153938] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [15988.929679] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16000.780743] loop0: detected capacity change from 0 to 204800000 [16000.973209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16000.988413] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16001.000033] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16006.593311] loop0: detected capacity change from 0 to 204800000 [16006.854155] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16006.864455] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16006.876460] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16012.256216] loop0: detected capacity change from 0 to 204800000 [16019.059785] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16019.303858] Lustre: fs15246-OST0000: new disk, initializing [16019.308163] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16019.315111] Lustre: 514148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16019.433832] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16026.173111] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [16038.279449] loop1: detected capacity change from 0 to 204800000 [16045.345057] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16045.479186] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16045.558057] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16045.659808] Lustre: fs15246-MDT0000: new disk, initializing [16045.805661] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16045.830851] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16045.952264] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16047.011243] Lustre: 515270:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000db33b8e x1875747912335232/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788852754 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16051.035525] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [16056.171255] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16056.204218] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16056.258787] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16077.801473] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16077.841407] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16082.851480] Lustre: server umount fs15246-MDT0000 complete [16082.940142] LustreError: 510550:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16088.050508] LustreError: 510550:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16093.188910] LustreError: 510549:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16096.048494] Lustre: server umount fs15246-OST0000 complete [16112.098261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16112.112189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16112.134264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16113.121458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16113.131363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16113.151380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16117.832868] Lustre: server umount lustre-OST0000 complete [16123.874375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16123.876199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16123.877122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16123.877127] Lustre: Skipped 1 previous similar message [16123.899919] Lustre: Skipped 1 previous similar message [16128.904555] Lustre: server umount lustre-MDT0000 complete [16128.995762] LustreError: 510549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16129.014432] LustreError: 510549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16133.274107] LustreError: 510533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852824 with bad export cookie 13358536971533128198 [16133.281385] LustreError: 510533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16133.282677] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16134.092387] Lustre: server umount lustre-MDT0001 complete [16152.115533] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [16156.680741] Key type lgssc unregistered [16157.026933] LNet: 518390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16157.032034] LNetError: 518390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16158.062386] LNet: Removed LNI 192.168.202.113@tcp [16159.237594] Key type .llcrypt unregistered [16159.239783] Key type ._llcrypt unregistered [16181.385737] Key type ._llcrypt registered [16181.390678] Key type .llcrypt registered [16181.535295] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [16183.243407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16183.307618] alg: No test for adler32 (adler32-zlib) [16184.367720] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [16184.728315] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [16186.480168] Key type lgssc registered [16188.166426] Lustre: Echo OBD driver; http://www.lustre.org/ [16234.501168] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [16247.709704] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [16290.651619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16290.685596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16292.089417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16292.113642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16292.187745] Lustre: lustre-MDT0000: new disk, initializing [16292.275167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16292.288468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16298.038403] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [16311.220448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16311.436329] Lustre: 522859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16311.524937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16311.532204] Lustre: Skipped 1 previous similar message [16311.637650] Lustre: lustre-MDT0001: new disk, initializing [16311.813378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16311.892684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16311.929479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16317.957602] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [16328.608269] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16334.942370] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16342.772376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16343.265313] Lustre: lustre-OST0000: new disk, initializing [16343.272944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16343.278244] Lustre: 524827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16343.384580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16349.876975] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [16350.884073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16350.892932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16351.038735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16360.632808] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16368.399662] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16368.669202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16373.818876] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16374.221361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16376.809207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16376.809940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16376.813363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16376.828910] Lustre: Skipped 1 previous similar message [16376.868342] Lustre: Skipped 1 previous similar message [16381.927069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16381.939911] Lustre: Skipped 1 previous similar message [16382.989876] Lustre: server umount lustre-OST0000 complete [16388.593903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16388.613531] LustreError: Skipped 1 previous similar message [16388.617368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16388.634596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16392.163044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16392.173268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16393.978571] Lustre: server umount lustre-MDT0000 complete [16397.924833] LustreError: 522851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853088 with bad export cookie 12356163833956614887 [16397.925812] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16397.930819] LustreError: 522851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16398.191294] Lustre: server umount lustre-MDT0001 complete [16426.267923] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [16439.011548] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [16482.423348] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16167 sec ========= 03:39:31 (1788853171) [16483.952138] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:39:34 (1788853174) === [16489.494933] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:39:38 (1788853178) === [16538.425839] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [16541.536400] Key type lgssc unregistered [16541.854537] LNet: 531478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16541.869331] LNetError: 531478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16541.880860] LNet: Removed LNI 192.168.202.113@tcp [16542.871188] Key type .llcrypt unregistered [16542.873679] Key type ._llcrypt unregistered