[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 853096873 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2624MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.004369] x2apic enabled [ 0.005009] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.008881] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.023568] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027757] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030022] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.033180] *** VALIDATE proc *** [ 0.034196] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.037021] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044242] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.046274] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055264] rcu: Hierarchical SRCU implementation. [ 0.057509] smp: Bringing up secondary CPUs ... [ 0.058643] x86: Booting SMP configuration: [ 0.059022] .... node #0, CPUs: #1 #2 #3 [ 0.071023] smp: Brought up 1 node, 4 CPUs [ 0.073009] smpboot: Max logical packages: 1 [ 0.074010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.116387] node 0 deferred pages initialised in 40ms [ 0.118167] devtmpfs: initialized [ 0.120152] x86/mm: Memory block size: 128MB [ 0.123299] gcov: version magic: 0x41383552 [ 0.125388] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127269] pinctrl core: initialized pinctrl subsystem [ 0.128207] [ 0.129009] ************************************************************* [ 0.130014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131014] ** ** [ 0.132011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133012] ** ** [ 0.134012] ** This means that this kernel is built to expose internal ** [ 0.135017] ** IOMMU data structures, which may compromise security on ** [ 0.136016] ** your system. ** [ 0.137018] ** ** [ 0.138014] ** If you see this message and you are not debugging the ** [ 0.139014] ** kernel, report this immediately to your vendor! ** [ 0.140016] ** ** [ 0.141015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142015] ************************************************************* [ 0.143695] NET: Registered protocol family 16 [ 0.144439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.148095] cpuidle: using governor menu [ 0.149762] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.152505] PCI: Using configuration type 1 for base access [ 0.154192] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.163280] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.164034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.166087] cryptd: max_cpu_qlen set to 1000 [ 0.169248] ACPI: Added _OSI(Module Device) [ 0.171015] ACPI: Added _OSI(Processor Device) [ 0.172009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.174016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.180059] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187007] ACPI: Interpreter enabled [ 0.189059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.191011] ACPI: Using IOAPIC for interrupt routing [ 0.193134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197919] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211699] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.217017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.226671] acpiphp: Slot [2] registered [ 0.228112] acpiphp: Slot [5] registered [ 0.230168] acpiphp: Slot [6] registered [ 0.231206] acpiphp: Slot [7] registered [ 0.233113] acpiphp: Slot [8] registered [ 0.235153] acpiphp: Slot [9] registered [ 0.236092] acpiphp: Slot [10] registered [ 0.238101] acpiphp: Slot [3] registered [ 0.239102] acpiphp: Slot [4] registered [ 0.241112] acpiphp: Slot [11] registered [ 0.242092] acpiphp: Slot [12] registered [ 0.244104] acpiphp: Slot [13] registered [ 0.246155] acpiphp: Slot [14] registered [ 0.247105] acpiphp: Slot [15] registered [ 0.249105] acpiphp: Slot [16] registered [ 0.251239] acpiphp: Slot [17] registered [ 0.252131] acpiphp: Slot [18] registered [ 0.254116] acpiphp: Slot [19] registered [ 0.256194] acpiphp: Slot [20] registered [ 0.258193] acpiphp: Slot [21] registered [ 0.259099] acpiphp: Slot [22] registered [ 0.261105] acpiphp: Slot [23] registered [ 0.263118] acpiphp: Slot [24] registered [ 0.265127] acpiphp: Slot [25] registered [ 0.266089] acpiphp: Slot [26] registered [ 0.268098] acpiphp: Slot [27] registered [ 0.269082] acpiphp: Slot [28] registered [ 0.271093] acpiphp: Slot [29] registered [ 0.272131] acpiphp: Slot [30] registered [ 0.274156] acpiphp: Slot [31] registered [ 0.276084] PCI host bridge to bus 0000:00 [ 0.277017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.285021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.288022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.291023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293259] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.296081] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.300472] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310925] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.316563] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.319016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327677] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330904] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338888] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.345014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.366020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372316] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453893] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.551021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.602839] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.615013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.625012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.646012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.655238] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.662014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.692018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703856] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.711012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.717015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.733000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.739975] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.740396] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.742354] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744412] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.750034] iommu: Default domain type: Passthrough [ 0.751553] SCSI subsystem initialized [ 0.753106] ACPI: bus type USB registered [ 0.755146] usbcore: registered new interface driver usbfs [ 0.756053] usbcore: registered new interface driver hub [ 0.758111] usbcore: registered new device driver usb [ 0.760147] pps_core: LinuxPPS API ver. 1 registered [ 0.761007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.764047] PTP clock support registered [ 0.766192] EDAC MC: Ver: 3.0.0 [ 0.768153] PCI: Using ACPI for IRQ routing [ 0.769000] NetLabel: Initializing [ 0.770015] NetLabel: domain hash size = 128 [ 0.771000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.774131] NetLabel: unlabeled traffic allowed by default [ 0.776250] vgaarb: loaded [ 0.777287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.789204] clocksource: Switched to clocksource kvm-clock [ 0.909211] VFS: Disk quotas dquot_6.6.0 [ 0.910652] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.913287] *** VALIDATE ramfs *** [ 0.914604] *** VALIDATE hugetlbfs *** [ 0.916195] pnp: PnP ACPI init [ 0.918461] pnp: PnP ACPI: found 6 devices [ 0.934296] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.937736] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.940925] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.943553] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.946221] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.949016] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.977253] NET: Registered protocol family 2 [ 0.980260] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.985110] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.988786] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.993987] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.997075] TCP: Hash tables configured (established 65536 bind 65536) [ 0.999958] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.003575] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006797] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.010290] NET: Registered protocol family 1 [ 1.013267] RPC: Registered named UNIX socket transport module. [ 1.015800] RPC: Registered udp transport module. [ 1.019981] RPC: Registered tcp transport module. [ 1.022758] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.025790] NET: Registered protocol family 44 [ 1.027799] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.030994] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035263] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.038738] PCI: CLS 0 bytes, default 64 [ 1.042615] Unpacking initramfs... [ 3.655347] debug: unmapping init [mem 0xffffa0d53cc54000-0xffffa0d53ffbffff] [ 3.660111] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.662994] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.666739] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.190552] Initialise system trusted keyrings [ 4.191956] Key type blacklist registered [ 4.193480] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.201744] zbud: loaded [ 4.204314] *** VALIDATE nfs *** [ 4.205604] *** VALIDATE nfs4 *** [ 4.207182] pstore: using deflate compression [ 4.210968] Platform Keyring initialized [ 4.339651] NET: Registered protocol family 38 [ 4.341681] Key type asymmetric registered [ 4.342983] Asymmetric key parser 'x509' registered [ 4.344698] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.347630] io scheduler mq-deadline registered [ 4.349397] io scheduler kyber registered [ 4.350899] io scheduler bfq registered [ 4.352675] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.355645] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.358425] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.361353] ACPI: Power Button [PWRF] [ 4.366592] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.372828] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.401870] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.413481] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.441979] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.471388] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.502227] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.507479] Non-volatile memory driver v1.3 [ 4.509410] Linux agpgart interface v0.103 [ 4.539725] virtio_blk virtio1: [vda] 146744 512-byte logical blocks (75.1 MB/71.7 MiB) [ 4.542867] vda: detected capacity change from 0 to 75132928 [ 4.557551] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.560411] vdb: detected capacity change from 0 to 1073741824 [ 4.574586] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.578184] vdc: detected capacity change from 0 to 2621440000 [ 4.594427] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.597196] vdd: detected capacity change from 0 to 2621440000 [ 4.612386] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.615701] vde: detected capacity change from 0 to 4294967296 [ 4.630477] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.634044] vdf: detected capacity change from 0 to 4294967296 [ 4.640980] libphy: Fixed MDIO Bus: probed [ 4.657376] usbcore: registered new interface driver usbserial_generic [ 4.660476] usbserial: USB Serial support registered for generic [ 4.663422] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.669087] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.671154] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.673786] mousedev: PS/2 mouse device common for all mice [ 4.677766] rtc_cmos 00:05: RTC can wake from S4 [ 4.681263] rtc_cmos 00:05: registered as rtc0 [ 4.682834] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.687112] intel_pstate: CPU model not supported [ 4.690690] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.693418] hid: raw HID events driver (C) Jiri Kosina [ 4.701152] usbcore: registered new interface driver usbhid [ 4.705230] usbhid: USB HID core driver [ 4.706542] drop_monitor: Initializing network drop monitor service [ 4.710509] Initializing XFRM netlink socket [ 4.713643] NET: Registered protocol family 10 [ 4.719168] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.720825] Segment Routing with IPv6 [ 4.730068] NET: Registered protocol family 17 [ 4.733669] mpls_gso: MPLS GSO support [ 4.740450] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.746142] RAS: Correctable Errors collector initialized. [ 4.748491] AVX version of gcm_enc/dec engaged. [ 4.750435] AES CTR mode by8 optimization enabled [ 4.840842] sched_clock: Marking stable (4840721994, 0)->(6044438530, -1203716536) [ 4.844888] registered taskstats version 1 [ 4.846773] Loading compiled-in X.509 certificates [ 4.848770] zswap: loaded using pool lzo/zbud [ 4.872858] Key type big_key registered [ 4.891594] Key type encrypted registered [ 4.893667] ima: No TPM chip found, activating TPM-bypass! [ 4.897746] ima: Allocated hash algorithm: sha1 [ 4.902396] ima: No architecture policies found [ 4.905807] evm: Initialising EVM extended attributes: [ 4.910511] evm: security.selinux [ 4.913423] evm: security.ima [ 4.915605] evm: security.capability [ 4.918968] evm: HMAC attrs: 0x1 [ 4.925527] rtc_cmos 00:05: setting system clock to 2026-09-08 23:03:04 UTC (1788908584) [ 4.938959] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 4.942537] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 4.952177] Write protecting the kernel read-only data: 28672k [ 4.956367] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 4.959705] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 4.995431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.005691] systemd[1]: Detected virtualization kvm. [ 5.007541] systemd[1]: Detected architecture x86-64. [ 5.009733] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.045726] systemd[1]: No hostname configured. [ 5.049895] systemd[1]: Set hostname to . [ 5.053943] random: systemd: uninitialized urandom read (16 bytes read) [ 5.058024] systemd[1]: Initializing machine ID from random generator. [ 5.244767] random: ln: uninitialized urandom read (6 bytes read) [ 5.320430] random: systemd: uninitialized urandom read (16 bytes read) [ 5.323451] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.392851] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.396586] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.849821] device-mapper: uevent: version 1.0.3 [ 6.852566] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.136382] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 9.366595] random: fast init done [ 9.925049] scsi host0: ata_piix [ 10.176262] scsi host1: ata_piix [ 10.178153] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.181570] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.488924] random: crng init done [ 14.490744] random: 7 urandom warning(s) missed due to ratelimiting [ 17.847234] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.218757] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.067459] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.196670] SELinux: Disabled at runtime. [ 22.326195] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.342363] systemd[1]: Detected virtualization kvm. [ 22.345327] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.721716] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.727714] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.745932] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.753501] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.759560] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.774723] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.801580] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Syste[ 24.119545] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ms. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.198784] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.112979] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.210604] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.799908] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.036541] EDAC sbridge: Ver: 1.1.2 [ 30.284365] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.210316] NFS: Registering the id_resolver key type [ 31.215357] Key type id_resolver registered [ 31.220711] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg334-server login: [ 73.969159] libcfs: loading out-of-tree module taints kernel. [ 74.001399] Key type ._llcrypt registered [ 74.003628] Key type .llcrypt registered [ 74.054710] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 84.202491] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 84.850611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.857298] alg: No test for adler32 (adler32-zlib) [ 85.990082] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 86.390440] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 88.024118] Key type lgssc registered [ 88.798208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.468094] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.439506] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 127.683975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.705398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.883206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 128.908028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 128.953893] Lustre: lustre-MDT0000: new disk, initializing [ 129.015081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.026102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.269726] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.572659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.621099] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 138.640836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 138.643688] Lustre: Skipped 1 previous similar message [ 138.692471] Lustre: lustre-MDT0001: new disk, initializing [ 138.725556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 138.743063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 138.749222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 140.852161] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.742529] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 148.768568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 148.902556] Lustre: lustre-OST0000: new disk, initializing [ 148.906086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 148.909281] Lustre: 8423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 148.948130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.261459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 152.268158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 152.295901] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.322212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 160.591451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 160.698176] Lustre: lustre-OST0001: new disk, initializing [ 160.702126] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 160.705789] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 160.749169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.353940] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.915024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 166.921756] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 166.965260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 172.048610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.809507] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 181.831783] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing check_logdir /tmp/testlogs/ [ 184.759325] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing yml_node [ 187.395419] Lustre: DEBUG MARKER: Client: 2.17.57.86 [ 189.167400] Lustre: DEBUG MARKER: MDS: 2.17.57.86 [ 190.757534] Lustre: DEBUG MARKER: OSS: 2.17.57.86 [ 191.648706] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 19:06:10 EDT 2026 [ 201.565780] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 202.644790] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 218.084166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.085123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 218.093336] Lustre: Skipped 1 previous similar message [ 223.203637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 223.208786] Lustre: Skipped 6 previous similar messages [ 228.325277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 228.328563] Lustre: Skipped 3 previous similar messages [ 230.368416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 230.483948] Lustre: server umount lustre-MDT0000 complete [ 232.519892] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908812 with bad export cookie 3837256767799441581 [ 232.523507] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 232.533515] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 233.446891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 233.454041] Lustre: Skipped 3 previous similar messages [ 233.454260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 233.463838] Lustre: Skipped 1 previous similar message [ 237.473412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 237.477144] Lustre: Skipped 1 previous similar message [ 238.757381] Lustre: server umount lustre-MDT0001 complete [ 241.117666] Lustre: server umount lustre-OST0000 complete [ 243.151290] Lustre: server umount lustre-OST0001 complete [ 246.806223] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 252.588604] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 273.364800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.504611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 273.531842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 273.593484] Lustre: lustre-MDT0000: new disk, initializing [ 273.652322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.662780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 275.919380] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 282.577974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.683266] Lustre: 15034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 282.723812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 282.726766] Lustre: Skipped 1 previous similar message [ 282.781806] Lustre: lustre-MDT0001: new disk, initializing [ 282.860616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.898223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 282.914676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 285.564821] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 291.555509] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 294.801185] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 298.989248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.155263] Lustre: lustre-OST0000: new disk, initializing [ 299.158186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 299.162933] Lustre: 17001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 299.211277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 300.446652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 300.455972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 300.533971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 302.596522] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 308.221820] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 310.396583] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 310.515309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 312.750262] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 312.886117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 315.876956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.877581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 315.878015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 315.885319] Lustre: Skipped 1 previous similar message [ 315.899036] LustreError: Skipped 1 previous similar message [ 328.162037] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 328.260673] Lustre: server umount lustre-OST0000 complete [ 331.235940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.244946] Lustre: Skipped 1 previous similar message [ 336.355079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.361855] Lustre: Skipped 8 previous similar messages [ 336.434856] Lustre: server umount lustre-MDT0000 complete [ 338.498623] LustreError: 15829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908918 with bad export cookie 3837256767799443114 [ 338.501246] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.505839] LustreError: 15829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.705078] Lustre: server umount lustre-MDT0001 complete [ 341.095495] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:08:40 (1788908920) [ 342.491394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 343.645359] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:08:42 (1788908922) [ 347.565535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.783118] LustreError: 19411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.791314] LustreError: 19411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 347.820078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 350.143607] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 353.251846] LustreError: 19412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.091087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.561802] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 361.440467] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 364.553620] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 368.573579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 372.172536] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 376.486971] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 385.507117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 385.509574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.517658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 385.523206] Lustre: Skipped 1 previous similar message [ 386.623932] Lustre: server umount lustre-OST0000 complete [ 394.938278] Lustre: server umount lustre-MDT0000 complete [ 395.746373] LustreError: 20139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.757783] LustreError: 20139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 397.070923] LustreError: 19391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908976 with bad export cookie 3837256767799443996 [ 397.072147] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 397.076233] LustreError: 19391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 397.256242] Lustre: server umount lustre-MDT0001 complete [ 403.200868] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 404.853659] Key type lgssc unregistered [ 405.024988] LNet: 22937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.030294] LNetError: 22937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 405.039792] LNet: Removed LNI 192.168.203.134@tcp [ 405.585179] Key type .llcrypt unregistered [ 405.588772] Key type ._llcrypt unregistered [ 418.682345] Key type ._llcrypt registered [ 418.684556] Key type .llcrypt registered [ 418.743836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 419.346374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 419.355388] alg: No test for adler32 (adler32-zlib) [ 420.276579] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 420.448174] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 422.088204] Key type lgssc registered [ 422.699435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 427.465337] loop: module loaded [ 427.491231] loop0: detected capacity change from 0 to 204800000 [ 427.632761] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 427.641156] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 427.844254] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 431.110850] loop0: detected capacity change from 0 to 204800000 [ 431.201703] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 431.208796] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 431.215939] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 434.076507] loop0: detected capacity change from 0 to 204800000 [ 437.266110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 437.284975] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.460818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.498632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.560179] Lustre: lustre-MDT0000: new disk, initializing [ 438.625418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.645465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.324038] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 447.233967] loop1: detected capacity change from 0 to 204800000 [ 450.476234] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.679880] Lustre: lustre-OST07c6: new disk, initializing [ 450.683795] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 450.688150] Lustre: Skipped 1 previous similar message [ 450.692534] Lustre: 26095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.738502] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 454.078823] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 454.746785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 454.754761] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 454.792938] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 458.062518] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 458.067224] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 458.070758] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 458.074543] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 458.078631] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 458.082734] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 472.918732] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 472.925399] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 472.929786] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 472.933509] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 472.937431] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 472.941418] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 476.852822] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 476.861643] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 476.865389] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 476.871025] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.875265] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 476.878869] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.882565] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 476.887297] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.890747] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 476.894667] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 476.898175] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 476.905603] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.004449] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 480.010161] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 480.014570] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 480.018827] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.022530] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 480.027649] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.032067] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 480.040090] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.044523] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 480.048389] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 480.053580] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 480.058305] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.017876] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 487.023287] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 487.030170] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 487.034751] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 487.038437] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 487.042278] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 487.045842] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 487.049065] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 487.052486] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 487.056555] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 487.060275] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.063786] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 496.887267] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 496.893852] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 496.898805] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 496.903923] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 496.909132] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 496.916267] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 496.920275] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 496.922985] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 496.926482] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 496.931508] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 496.934876] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 496.939099] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 541.667457] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.670260] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.677318] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 544.354995] Lustre: server umount lustre-OST07c6 complete [ 548.767165] Lustre: server umount lustre-MDT0000 complete [ 555.957554] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 557.359537] Key type lgssc unregistered [ 557.520572] LNet: 28666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.528828] LNetError: 28666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.535607] LNet: Removed LNI 192.168.203.134@tcp [ 557.957256] Key type .llcrypt unregistered [ 557.959279] Key type ._llcrypt unregistered [ 567.066798] Key type ._llcrypt registered [ 567.068602] Key type .llcrypt registered [ 567.116160] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 567.519237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 567.554130] alg: No test for adler32 (adler32-zlib) [ 568.438511] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 568.544378] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 570.128143] Key type lgssc registered [ 570.654568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 574.253826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 574.264734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.497547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.374300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 581.159272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.347466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.188909] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 586.887508] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 589.427903] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 592.485249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.673148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.382408] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 598.474093] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 600.871930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 613.862526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 613.862553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.866082] LustreError: Skipped 1 previous similar message [ 613.871960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.417740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.418213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.422784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.513205] Lustre: server umount lustre-OST0000 complete [ 621.538694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.540296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.547595] Lustre: Skipped 1 previous similar message [ 621.550049] Lustre: Skipped 1 previous similar message [ 625.610397] Lustre: server umount lustre-MDT0000 complete [ 626.657466] LustreError: 29909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.669950] LustreError: 29909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 627.559364] LustreError: 31752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909207 with bad export cookie 13790031747285853345 [ 627.563408] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.568538] LustreError: 31752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.722428] Lustre: server umount lustre-MDT0001 complete [ 632.435985] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 633.647992] Key type lgssc unregistered [ 633.788684] LNet: 33644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.793245] LNetError: 33644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.807558] LNet: Removed LNI 192.168.203.134@tcp [ 634.170738] Key type .llcrypt unregistered [ 634.172708] Key type ._llcrypt unregistered [ 643.133289] Key type ._llcrypt registered [ 643.134521] Key type .llcrypt registered [ 643.177506] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 643.630582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.643931] alg: No test for adler32 (adler32-zlib) [ 644.526956] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 644.625358] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 646.216217] Key type lgssc registered [ 646.686806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.744894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.754969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.920867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.473984] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 655.808676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.027754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 657.649029] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 660.289516] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 662.635976] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.286657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.455149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.727267] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 670.626536] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 675.689246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 680.929583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.931501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 680.932553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 680.938434] Lustre: Skipped 1 previous similar message [ 682.146225] Lustre: server umount lustre-OST0000 complete [ 686.010845] Lustre: server umount lustre-MDT0000 complete [ 686.050817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.051947] LustreError: 34885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.057520] Lustre: Skipped 1 previous similar message [ 686.065294] LustreError: 34885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 687.635233] LustreError: 34916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909267 with bad export cookie 13620221926267416951 [ 687.638049] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.645489] LustreError: 34916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 687.783973] Lustre: server umount lustre-MDT0001 complete [ 692.644336] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 693.974100] Key type lgssc unregistered [ 694.133464] LNet: 38425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.137303] LNetError: 38425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.146470] LNet: Removed LNI 192.168.203.134@tcp [ 694.571335] Key type .llcrypt unregistered [ 694.573214] Key type ._llcrypt unregistered [ 706.043303] Key type ._llcrypt registered [ 706.044575] Key type .llcrypt registered [ 706.104921] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 706.543385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 706.574517] alg: No test for adler32 (adler32-zlib) [ 707.476415] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 707.584392] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 709.176526] Key type lgssc registered [ 709.708322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.475906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.483827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.706503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.779311] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 720.926158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.169182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 723.472140] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 727.088724] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 730.165611] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.943713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.121471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.138738] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 740.272440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 740.789196] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 752.097512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 752.101735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.107886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.610531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.616756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.622252] Lustre: Skipped 1 previous similar message [ 755.482301] Lustre: server umount lustre-MDT0000 complete [ 757.201469] LustreError: 39795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909336 with bad export cookie 18142915577453349459 [ 757.207396] LustreError: 39795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 757.207612] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 757.729140] LustreError: 42302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.733605] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.741043] LustreError: 42302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 757.755286] Lustre: Skipped 1 previous similar message [ 757.759633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 757.762114] Lustre: Skipped 1 previous similar message [ 758.579548] LustreError: 42302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.851029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 762.851046] LustreError: 39810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.854153] Lustre: Skipped 1 previous similar message [ 763.424724] Lustre: server umount lustre-MDT0001 complete [ 774.654182] LustreError: 39793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.34@tcp arrived at 1788909354 with bad export cookie 18142915577453350306 [ 774.663377] LustreError: 39793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 782.129079] Lustre: server umount lustre-OST0000 complete [ 790.778566] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 792.060625] Key type lgssc unregistered [ 792.233125] LNet: 43667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.237685] LNetError: 43667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.250500] LNet: Removed LNI 192.168.203.134@tcp [ 792.623196] Key type .llcrypt unregistered [ 792.625937] Key type ._llcrypt unregistered [ 803.995702] Key type ._llcrypt registered [ 803.997628] Key type .llcrypt registered [ 804.048947] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 804.513305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.523546] alg: No test for adler32 (adler32-zlib) [ 805.403188] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 805.507161] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 807.096148] Key type lgssc registered [ 807.551662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.162381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.173864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.382655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.150419] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 818.198477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.395629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.397565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 823.756432] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 826.476344] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 829.776777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.955827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.712329] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 836.127622] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 837.159993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 847.331641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.336253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.350419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 848.355875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 848.357147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 848.360933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.545266] Lustre: server umount lustre-OST0000 complete [ 864.737436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 864.740896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.747424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.750864] Lustre: Skipped 2 previous similar messages [ 868.833677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.834836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.373535] Lustre: server umount lustre-MDT0000 complete [ 872.081119] LustreError: 45036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909451 with bad export cookie 13010724276498591596 [ 872.083718] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.086232] LustreError: 45036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 878.295413] Lustre: server umount lustre-MDT0001 complete [ 883.252265] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 884.679176] Key type lgssc unregistered [ 884.854337] LNet: 48756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.859547] LNetError: 48756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.873807] LNet: Removed LNI 192.168.203.134@tcp [ 885.270620] Key type .llcrypt unregistered [ 885.272324] Key type ._llcrypt unregistered [ 896.077620] Key type ._llcrypt registered [ 896.079611] Key type .llcrypt registered [ 896.128091] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 896.499159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.525238] alg: No test for adler32 (adler32-zlib) [ 897.405547] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 897.508324] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 899.104156] Key type lgssc registered [ 899.630654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.068388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.077847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.279818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.075862] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 910.022115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.193822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.187279] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 915.427173] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 918.038714] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.985801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.163063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.753557] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 926.728360] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 929.383380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 939.146713] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 940.448951] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 951.333104] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0819h18m50s [ 951.977456] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 953.314738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.315893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.320348] Lustre: Skipped 2 previous similar messages [ 953.322573] Lustre: Skipped 1 previous similar message [ 954.338621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.341313] Lustre: Skipped 3 previous similar messages [ 960.482456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.485210] Lustre: Skipped 3 previous similar messages [ 965.872683] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 965.877611] Lustre: Skipped 1 previous similar message [ 967.136146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 967.209850] Lustre: server umount lustre-MDT0000 complete [ 968.660249] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 971.595853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.664719] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.789237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.823215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 972.840366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 973.588771] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 975.110170] Lustre: DEBUG MARKER: Wait for df (36578) ... [ 975.758484] Lustre: DEBUG MARKER: done [ 977.889971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.892909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.898043] Lustre: Skipped 1 previous similar message [ 977.901468] Lustre: Skipped 3 previous similar messages [ 988.131898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.134505] Lustre: Skipped 4 previous similar messages [ 991.200144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 991.270665] Lustre: server umount lustre-OST0000 complete [ 993.249926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 993.249962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.258157] Lustre: Skipped 1 previous similar message [ 999.015545] Lustre: server umount lustre-MDT0000 complete [ 999.329388] LustreError: 50149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.338126] LustreError: 50149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1000.583808] LustreError: 54332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909580 with bad export cookie 4655353675548396332 [ 1000.586106] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.588266] LustreError: 54332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1000.727856] Lustre: server umount lustre-MDT0001 complete [ 1004.995906] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1006.221600] Key type lgssc unregistered [ 1006.381504] LNet: 55083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.385175] LNetError: 55083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.394593] LNet: Removed LNI 192.168.203.134@tcp [ 1006.778986] Key type .llcrypt unregistered [ 1006.780277] Key type ._llcrypt unregistered [ 1024.676742] Key type ._llcrypt registered [ 1024.678208] Key type .llcrypt registered [ 1024.725378] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1025.112910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.128113] alg: No test for adler32 (adler32-zlib) [ 1026.041485] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1026.179793] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1027.784141] Key type lgssc registered [ 1028.326090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.853283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.866357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.990075] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1033.002456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1033.091713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.868980] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1038.746815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.802441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1038.826775] Lustre: 57262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1038.963075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.715874] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1043.937277] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1046.465178] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1049.210978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.306293] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1049.390393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.840800] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1054.918275] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1056.314546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1064.318666] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0819h20m43s [ 1065.050343] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1068.289266] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1069.456231] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1069.483343] LustreError: 57269:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1089.861285] Lustre: lustre-MDT0000: Client 854161b5-70ac-44a7-bf27-fca454a347c5 (at 192.168.203.34@tcp) reconnecting [ 1109.536212] LustreError: 57269:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1110.547292] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1111.748784] Lustre: DEBUG MARKER: done [ 1115.109426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1115.113928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.124315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.666911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.668311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.674727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.181695] Lustre: server umount lustre-OST0000 complete [ 1122.786614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.787947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.796487] Lustre: Skipped 1 previous similar message [ 1122.803952] Lustre: Skipped 2 previous similar messages [ 1128.121590] Lustre: server umount lustre-MDT0000 complete [ 1130.071374] LustreError: 58005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909709 with bad export cookie 2282069213634086118 [ 1130.075308] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.080947] LustreError: 58005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.249992] Lustre: server umount lustre-MDT0001 complete [ 1135.004271] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1136.383949] Key type lgssc unregistered [ 1136.531948] LNet: 61583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.535666] LNetError: 61583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.547935] LNet: Removed LNI 192.168.203.134@tcp [ 1136.990589] Key type .llcrypt unregistered [ 1136.993182] Key type ._llcrypt unregistered [ 1157.168309] Key type ._llcrypt registered [ 1157.169821] Key type .llcrypt registered [ 1157.220613] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1157.542762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.588882] alg: No test for adler32 (adler32-zlib) [ 1158.472363] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1158.586129] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1160.184220] Key type lgssc registered [ 1160.693557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.115554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.127111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.239546] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1163.248955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1163.317400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.686309] Lustre: Failing over lustre-MDT0000 [ 1163.797888] Lustre: server umount lustre-MDT0000 complete [ 1166.370124] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:22:25 (1788909745) [ 1169.077254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.287141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.204463] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1174.903127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.968145] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1174.984787] Lustre: 64353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1175.108337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1176.971406] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1180.011045] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1182.171020] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1184.939225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1185.035095] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1185.126954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.584904] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1190.695506] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1191.450829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1199.341151] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1201.635116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.638185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.641436] Lustre: Skipped 2 previous similar messages [ 1201.646540] Lustre: Skipped 2 previous similar messages [ 1206.754304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.756862] Lustre: Skipped 2 previous similar messages [ 1207.520672] Lustre: server umount lustre-MDT0000 complete [ 1208.976379] LustreError: 65087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909788 with bad export cookie 7496582949482636248 [ 1208.979660] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.982421] LustreError: 65087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1209.131385] Lustre: server umount lustre-MDT0001 complete [ 1210.441938] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1212.198234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1215.442956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.040698] LustreError: 63275:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f3be422b x1875807817467008/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1219.226490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.279260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1220.790522] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1224.348862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.500405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.152837] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1227.616143] Lustre: 63277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909791/real 1788909791] req@000000003ba1acc6 x1875807817466624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1229.190454] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1229.800399] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1230.819716] Lustre: 63278:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909793/real 1788909793] req@0000000065d020c4 x1875807817466752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1231.571540] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1231.845548] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1232.864155] Lustre: 63279:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909796/real 1788909796] req@00000000c66e40d7 x1875807817466880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1234.918423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1234.922884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.940064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.936137] Lustre: 63276:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909798/real 1788909798] req@0000000080660933 x1875807817467136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1236.968357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.968556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.979021] Lustre: Skipped 1 previous similar message [ 1236.982374] Lustre: Skipped 1 previous similar message [ 1240.128644] Lustre: server umount lustre-MDT0000 complete [ 1241.827108] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909821 with bad export cookie 7496582949482638516 [ 1241.829581] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.835586] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1241.980804] Lustre: server umount lustre-MDT0001 complete [ 1244.717037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1248.730979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.320655] LustreError: 63275:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000066a9f94f x1875807817486592/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1252.452377] LustreError: 66199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.462172] LustreError: 66199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1252.491725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.494770] Lustre: Skipped 1 previous similar message [ 1252.519154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1254.307580] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1256.613995] LustreError: 68848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.617264] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1256.619200] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.632286] Lustre: 63278:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909821/real 1788909821] req@00000000e0d12e5d x1875807817486080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1258.232417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.262248] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1263.591119] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1263.995443] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.455122] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1267.680187] Lustre: 63277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788909831/real 1788909831] req@000000004fe6f3fe x1875807817486464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788909847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1267.689502] Lustre: 63277:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1273.826210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.827717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.829272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.829277] Lustre: Skipped 1 previous similar message [ 1273.841245] Lustre: Skipped 2 previous similar messages [ 1275.047994] Lustre: server umount lustre-MDT0000 complete [ 1276.480983] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909856 with bad export cookie 7496582949482639895 [ 1276.484593] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.486375] LustreError: 64345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1276.591936] Lustre: server umount lustre-MDT0001 complete [ 1279.141960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1280.650864] Lustre: DEBUG MARKER: files compared the same [ 1291.817562] Lustre: server umount lustre-OST0000 complete [ 1298.169836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1299.323160] Key type lgssc unregistered [ 1299.458083] LNet: 74064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.462731] LNetError: 74064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.471387] LNet: Removed LNI 192.168.203.134@tcp [ 1299.839084] Key type .llcrypt unregistered [ 1299.840473] Key type ._llcrypt unregistered [ 1309.928938] Key type ._llcrypt registered [ 1309.930462] Key type .llcrypt registered [ 1309.977755] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1310.367163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.417812] alg: No test for adler32 (adler32-zlib) [ 1311.292162] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1311.399361] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1312.992229] Key type lgssc registered [ 1313.448306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.451614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.461546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.627673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.095617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 1322.108447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.229864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1324.057414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 1327.129600] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.496717] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1332.096506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.208631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.766800] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 1338.004232] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1343.462966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1351.810762] Lustre: server umount lustre-OST0000 complete [ 1356.768875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.769806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.774364] Lustre: Skipped 1 previous similar message [ 1356.778468] Lustre: Skipped 1 previous similar message [ 1361.889176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.893523] Lustre: Skipped 1 previous similar message [ 1363.959148] Lustre: server umount lustre-MDT0000 complete [ 1365.414991] LustreError: 77950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909944 with bad export cookie 17748834774124762863 [ 1365.416858] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.420571] LustreError: 77950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1375.647123] Lustre: server umount lustre-MDT0001 complete [ 1380.067933] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1381.283195] Key type lgssc unregistered [ 1381.427861] LNet: 79004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.433921] LNetError: 79004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.443185] LNet: Removed LNI 192.168.203.134@tcp [ 1381.833261] Key type .llcrypt unregistered [ 1381.834912] Key type ._llcrypt unregistered [ 1392.151054] Key type ._llcrypt registered [ 1392.152709] Key type .llcrypt registered [ 1392.198432] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1392.558766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1392.608451] alg: No test for adler32 (adler32-zlib) [ 1393.496659] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1393.598577] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1395.184161] Key type lgssc registered [ 1395.645131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1398.885119] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1398.900569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.368893] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1419.488417] LustreError: 80366:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 1446.112191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.149741] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1450.579961] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1453.276192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.320115] Lustre: *** cfs_fail_loc=706, val=0*** [ 1453.446993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.469348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1454.920070] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1457.792839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.949437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1459.477386] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1462.035634] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1464.012724] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1466.341509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.343333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.347766] Lustre: Skipped 1 previous similar message [ 1471.459017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1471.461314] Lustre: Skipped 2 previous similar messages [ 1476.578300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.582463] Lustre: Skipped 1 previous similar message [ 1479.136130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1479.198313] Lustre: server umount lustre-OST0000 complete [ 1481.698338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.704194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.706341] Lustre: Skipped 1 previous similar message [ 1486.682810] Lustre: server umount lustre-MDT0000 complete [ 1486.817225] LustreError: 81455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.822687] LustreError: 81455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1488.021236] LustreError: 80373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910067 with bad export cookie 4916731211135186958 [ 1488.024932] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.025533] LustreError: 80373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1488.148406] Lustre: server umount lustre-MDT0001 complete [ 1491.853844] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1492.907079] Key type lgssc unregistered [ 1493.037423] LNet: 83977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.040833] LNetError: 83977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.050425] LNet: Removed LNI 192.168.203.134@tcp [ 1493.423259] Key type .llcrypt unregistered [ 1493.424665] Key type ._llcrypt unregistered [ 1504.472045] Key type ._llcrypt registered [ 1504.473287] Key type .llcrypt registered [ 1504.519887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1504.893264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1504.934347] alg: No test for adler32 (adler32-zlib) [ 1505.808620] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1505.902242] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1507.488339] Key type lgssc registered [ 1507.884494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.704981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.712506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.200751] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1515.892795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.140607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1517.633603] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1520.416819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1520.541055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.653290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1525.668071] LustreError: 86054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.677476] LustreError: 86054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 1526.183518] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.192121] Lustre: MGS: Client 68ae7b47-aafd-4e3e-8dfc-a7ff3b765f1a (at 0@lo) reconnecting [ 1526.207346] LustreError: 86065:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1526.326089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.375414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1527.847184] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1536.483591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.490530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.490538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1541.604262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.607333] Lustre: Skipped 1 previous similar message [ 1542.108499] Lustre: server umount lustre-OST0000 complete [ 1546.722566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1546.722894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.723912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.723920] Lustre: Skipped 1 previous similar message [ 1546.734623] Lustre: Skipped 2 previous similar messages [ 1549.722474] Lustre: server umount lustre-MDT0000 complete [ 1551.394993] Lustre: server umount lustre-MDT0001 complete [ 1552.922226] Lustre: server umount MGS complete [ 1556.921708] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:28:56 (1788910136) [ 1566.695579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1569.550022] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1581.559827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.921697] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1586.862564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.946918] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1586.961878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1586.997356] Lustre: lustre-MDT0001: new disk, initializing [ 1587.022580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.033578] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1587.037070] Lustre: 93206:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1587.041409] LustreError: 93206:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1588.462449] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1592.785073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.886161] Lustre: lustre-OST0000: new disk, initializing [ 1592.888836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1592.892592] Lustre: 93973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1592.916255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.919931] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1598.782540] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1599.520750] Lustre: 94635:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bf1ef426 x1875808181323392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788910195 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1601.568964] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.577720] Lustre: MGS: Client 351ad0e7-d7d8-4397-a910-9a2cc202be56 (at 0@lo) reconnecting [ 1601.589723] LustreError: 93191:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1601.613254] Lustre: 92475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1601.644941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1601.691814] Lustre: lustre-MDT0000: new disk, initializing [ 1601.764197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.780924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1602.849053] Lustre: 95040:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000082680c9d x1875808181328896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788910198 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1602.849275] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1603.335727] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1607.908366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1607.911566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1607.915787] Lustre: Skipped 1 previous similar message [ 1607.924700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1628.640805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.643195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.648447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.650522] Lustre: Skipped 1 previous similar message [ 1633.231055] Lustre: server umount lustre-OST0000 complete [ 1636.833539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.833895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.836153] LustreError: Skipped 1 previous similar message [ 1636.839490] Lustre: Skipped 1 previous similar message [ 1636.840089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.844394] Lustre: Skipped 4 previous similar messages [ 1640.518693] Lustre: server umount lustre-MDT0000 complete [ 1656.288102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1656.369960] Lustre: server umount lustre-MDT0001 complete [ 1657.600253] Lustre: server umount MGS complete [ 1661.536402] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:30:40 (1788910240) [ 1667.476162] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1668.586895] Key type lgssc unregistered [ 1668.718351] LNet: 97535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.722451] LNetError: 97535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.731416] LNet: Removed LNI 192.168.203.134@tcp [ 1669.074568] Key type .llcrypt unregistered [ 1669.076309] Key type ._llcrypt unregistered [ 1676.172339] Key type ._llcrypt registered [ 1676.173297] Key type .llcrypt registered [ 1676.209412] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1676.545451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.552995] alg: No test for adler32 (adler32-zlib) [ 1677.427662] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1677.514474] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1679.096164] Key type lgssc registered [ 1679.491923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1682.331710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1682.340908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.510065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.047974] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1688.029245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.154721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.526040] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1692.207207] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1694.048184] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1696.561489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1696.689984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.705608] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1701.497281] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1708.964625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1718.240992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.244703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.250176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.267366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.267375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.268178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.730449] Lustre: server umount lustre-OST0000 complete [ 1723.873653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1723.876643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.884394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.889853] Lustre: Skipped 1 previous similar message [ 1728.350688] Lustre: server umount lustre-MDT0000 complete [ 1729.504964] LustreError: 98728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.510473] LustreError: 98728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1729.762404] LustreError: 98709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910309 with bad export cookie 5995315475122053585 [ 1729.765498] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.769221] LustreError: 98709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1729.899569] Lustre: server umount lustre-MDT0001 complete [ 1733.850795] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1734.971606] Key type lgssc unregistered [ 1735.103721] LNet: 102278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.106608] LNetError: 102278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.114459] LNet: Removed LNI 192.168.203.134@tcp [ 1735.449381] Key type .llcrypt unregistered [ 1735.450928] Key type ._llcrypt unregistered [ 1742.434877] Key type ._llcrypt registered [ 1742.436464] Key type .llcrypt registered [ 1742.481468] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1742.850850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.876095] alg: No test for adler32 (adler32-zlib) [ 1743.745382] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1743.855837] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1745.440193] Key type lgssc registered [ 1745.891730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.239396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.239513] LustreError: 103336:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1748.242520] Lustre: Skipped 1 previous similar message [ 1748.247241] LustreError: 103337:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1748.250453] LustreError: 103336:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1748.254081] LustreError: 103336:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1748.259812] LustreError: 103336:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1748.260750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.454303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.162943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.281590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.782501] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1755.553616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lsmod [ 1756.562105] LustreError: 104540:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1756.566266] LustreError: 104541:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1756.570050] LustreError: 104540:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1756.573029] LustreError: 104540:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1756.577812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.579136] LustreError: 104540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1756.697568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.761736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.763272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.767977] Lustre: Skipped 2 previous similar messages [ 1761.770759] Lustre: Skipped 1 previous similar message [ 1762.786344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.789528] Lustre: Skipped 2 previous similar messages [ 1763.870914] Lustre: server umount lustre-MDT0000 complete [ 1765.373773] LustreError: 103356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910344 with bad export cookie 15490975287541711873 [ 1765.377025] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.379459] LustreError: 103356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1766.880310] LustreError: 103805:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f4c0ea65 x1875808430865408/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1766.937736] Lustre: server umount lustre-MDT0001 complete [ 1777.505462] Lustre: server umount lustre-OST0000 complete [ 1780.436369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.604434] LustreError: 105603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.611896] LustreError: 105603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1780.629377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.067777] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1785.108245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.231482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1786.705820] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1789.398444] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1791.325209] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.743131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.880082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.103563] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1798.876790] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1807.211033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1812.450206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.452295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1812.452981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.454954] Lustre: Skipped 1 previous similar message [ 1816.936200] Lustre: server umount lustre-OST0000 complete [ 1821.152857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1821.155667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.160628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.162445] Lustre: Skipped 1 previous similar message [ 1824.543260] Lustre: server umount lustre-MDT0000 complete [ 1825.889711] LustreError: 105585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910405 with bad export cookie 15490975287541712538 [ 1825.892470] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1825.894330] LustreError: 105585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1826.014078] Lustre: server umount lustre-MDT0001 complete [ 1829.727989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1830.852204] Key type lgssc unregistered [ 1830.986242] LNet: 109152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.990124] LNetError: 109152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.997645] LNet: Removed LNI 192.168.203.134@tcp [ 1831.367583] Key type .llcrypt unregistered [ 1831.368920] Key type ._llcrypt unregistered [ 1840.702700] Key type ._llcrypt registered [ 1840.704415] Key type .llcrypt registered [ 1840.746581] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1841.140353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.181739] alg: No test for adler32 (adler32-zlib) [ 1842.036561] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1842.131689] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1843.712205] Key type lgssc registered [ 1844.120595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1846.949088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1846.959261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.137340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.578248] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1852.945402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.090061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1854.578150] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1857.267190] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1859.321718] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1861.754948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.919969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.190853] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1866.873043] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1873.192452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 1879.521458] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1880.536285] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1883.621364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.624213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.626812] LustreError: Skipped 1 previous similar message [ 1883.635336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.638707] Lustre: Skipped 1 previous similar message [ 1887.961604] Lustre: server umount lustre-OST0000 complete [ 1893.858132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.858802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.862927] Lustre: Skipped 2 previous similar messages [ 1895.449402] Lustre: server umount lustre-MDT0000 complete [ 1896.807434] LustreError: 110523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910476 with bad export cookie 4625430972498817302 [ 1896.809255] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.812156] LustreError: 110523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1896.939325] Lustre: server umount lustre-MDT0001 complete [ 1900.854425] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1901.950274] Key type lgssc unregistered [ 1902.084518] LNet: 114195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.087604] LNetError: 114195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.094489] LNet: Removed LNI 192.168.203.134@tcp [ 1902.433080] Key type .llcrypt unregistered [ 1902.434634] Key type ._llcrypt unregistered [ 1909.808533] Key type ._llcrypt registered [ 1909.809585] Key type .llcrypt registered [ 1909.851976] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1910.216257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1910.274449] alg: No test for adler32 (adler32-zlib) [ 1911.153517] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1911.249615] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1912.832330] Key type lgssc registered [ 1913.202815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1915.921221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1915.929907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.074446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.509431] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1921.559287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.699201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.083936] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1925.497980] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.298540] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.488382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.599083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.545863] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1933.958110] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1941.927132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1947.107619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1947.107936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.115344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.117402] Lustre: Skipped 1 previous similar message [ 1949.012624] Lustre: server umount lustre-OST0000 complete [ 1952.226059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.227297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.232443] Lustre: Skipped 2 previous similar messages [ 1952.404655] Lustre: server umount lustre-MDT0000 complete [ 1953.629043] LustreError: 115370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910533 with bad export cookie 7563163708813375224 [ 1953.631714] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.632721] LustreError: 115370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1953.749185] Lustre: server umount lustre-MDT0001 complete [ 1957.275937] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1958.424793] Key type lgssc unregistered [ 1958.548838] LNet: 118935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.552307] LNetError: 118935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.566798] LNet: Removed LNI 192.168.203.134@tcp [ 1958.908421] Key type .llcrypt unregistered [ 1958.910888] Key type ._llcrypt unregistered [ 1968.331598] Key type ._llcrypt registered [ 1968.332689] Key type .llcrypt registered [ 1968.368890] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1968.708149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1968.732635] alg: No test for adler32 (adler32-zlib) [ 1969.591333] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 1969.685987] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1971.264148] Key type lgssc registered [ 1971.633666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1974.205904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1974.212807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.346996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.699224] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1979.634497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.799076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1981.212674] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1983.647874] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1985.485041] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.758679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.869228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.824763] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1992.119759] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2000.172477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2005.211033] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2006.451090] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2006.453945] Lustre: Skipped 2 previous similar messages [ 2007.911760] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2007.914863] Lustre: Skipped 2 previous similar messages [ 2014.718441] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2014.731655] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2016.805246] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2016.810238] Lustre: Skipped 2 previous similar messages [ 2024.967951] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp [ 2026.461207] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.34/24@tcp in log lustre-MDT0000 [ 2026.466085] Lustre: Skipped 2 previous similar messages [ 2034.175439] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 2034.180030] Lustre: Skipped 1 previous similar message [ 2040.805626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2040.806083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.809206] LustreError: Skipped 2 previous similar messages [ 2040.815098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2041.313706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2041.316475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2041.316604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.317433] LustreError: Skipped 2 previous similar messages [ 2041.320040] Lustre: Skipped 1 previous similar message [ 2042.577677] Lustre: server umount lustre-OST0000 complete [ 2046.433693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2046.433894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.434529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.444196] Lustre: Skipped 1 previous similar message [ 2050.136859] Lustre: server umount lustre-MDT0000 complete [ 2051.393205] LustreError: 120307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910630 with bad export cookie 1484862659893003133 [ 2051.396250] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.397248] LustreError: 120307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2051.518524] Lustre: server umount lustre-MDT0001 complete [ 2055.333775] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2056.414954] Key type lgssc unregistered [ 2056.551529] LNet: 125706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.554823] LNetError: 125706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.562323] LNet: Removed LNI 192.168.203.134@tcp [ 2056.879367] Key type .llcrypt unregistered [ 2056.880422] Key type ._llcrypt unregistered [ 2068.073986] Key type ._llcrypt registered [ 2068.075351] Key type .llcrypt registered [ 2068.115765] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2068.437242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.486066] alg: No test for adler32 (adler32-zlib) [ 2069.345948] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2069.435026] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2071.024142] Key type lgssc registered [ 2071.386632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2074.111164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2074.119432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.316513] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2075.321773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.327120] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.34/24@tcp [ 2076.683405] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2079.729249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.876634] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2079.886133] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 2079.901055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.285997] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2083.905304] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2085.915602] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2088.434623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.554580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.580099] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2093.092204] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.817362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 2110.435656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.435704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.438761] LustreError: Skipped 1 previous similar message [ 2110.443437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.448073] Lustre: Skipped 1 previous similar message [ 2113.128446] Lustre: server umount lustre-OST0000 complete [ 2115.553774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.554367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.557605] Lustre: Skipped 2 previous similar messages [ 2120.673098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.675143] Lustre: Skipped 2 previous similar messages [ 2120.879245] Lustre: server umount lustre-MDT0000 complete [ 2122.285743] LustreError: 127223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910701 with bad export cookie 7369251119073706895 [ 2122.289986] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.290836] LustreError: 127223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.403596] Lustre: server umount lustre-MDT0001 complete [ 2126.678507] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2127.868665] Key type lgssc unregistered [ 2128.042513] LNet: 130846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.046284] LNetError: 130846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.058673] LNet: Removed LNI 192.168.203.134@tcp [ 2128.422617] Key type .llcrypt unregistered [ 2128.424290] Key type ._llcrypt unregistered [ 2137.706521] Key type ._llcrypt registered [ 2137.707928] Key type .llcrypt registered [ 2137.745907] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2148.144106] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2149.187574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.222150] alg: No test for adler32 (adler32-zlib) [ 2150.179924] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2150.521133] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2152.248363] Key type lgssc registered [ 2153.259644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2180.376111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.421806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.567942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2181.585840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2181.641513] Lustre: lustre-MDT0000: new disk, initializing [ 2181.694307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.709698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2185.647352] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2194.695242] hrtimer: interrupt took 3531306 ns [ 2197.036201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.201681] Lustre: 134218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2197.256293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2197.260130] Lustre: Skipped 1 previous similar message [ 2197.318632] Lustre: lustre-MDT0001: new disk, initializing [ 2197.477093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.541985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2197.565530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2202.100456] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2212.301674] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.145259] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.430189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2222.630696] Lustre: lustre-OST0000: new disk, initializing [ 2222.633313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2222.638812] Lustre: 136185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2222.721429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.275491] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2232.349687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2232.356062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2232.408048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2232.985645] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2235.213772] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2235.365226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2237.697123] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2237.814211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2242.532400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.533453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.536733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2242.543669] Lustre: Skipped 1 previous similar message [ 2247.657576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.660353] Lustre: Skipped 2 previous similar messages [ 2252.773427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.777235] Lustre: Skipped 1 previous similar message [ 2253.280784] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2253.365996] Lustre: server umount lustre-OST0000 complete [ 2257.891074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.905306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.909340] Lustre: Skipped 1 previous similar message [ 2263.615642] Lustre: server umount lustre-MDT0000 complete [ 2266.337668] LustreError: 134211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910845 with bad export cookie 10736460435219198542 [ 2266.342593] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.345957] LustreError: 134211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2266.619249] Lustre: server umount lustre-MDT0001 complete [ 2272.818385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.142425] LustreError: 138399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.162596] LustreError: 138399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2273.206766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.318286] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2278.372988] LustreError: 138400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.743884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.107510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.194257] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2290.283406] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2293.655725] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.121038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.397729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.075820] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2306.473442] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.993881] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2309.090675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2311.765880] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2311.929413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2317.800436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.904741] Lustre: lustre-OST0001: new disk, initializing [ 2317.908739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2317.913738] Lustre: 141769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2317.981473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.168484] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2323.503142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2323.509696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2323.579875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2326.025492] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2326.169426] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2328.642756] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2328.770352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2331.962727] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bf049b28800.ost_server_uuid 50 [ 2332.791282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bf049b28800.ost_server_uuid in FULL state after 0 sec [ 2338.795608] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2338.798122] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.801694] LustreError: Skipped 1 previous similar message [ 2338.803495] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2338.816873] Lustre: Skipped 2 previous similar messages [ 2343.117310] Lustre: server umount lustre-OST0001 complete [ 2343.908655] LustreError: 140218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.919836] LustreError: 140218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2345.443184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.448515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.491044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.495659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2347.497946] Lustre: Skipped 2 previous similar messages [ 2349.030177] LustreError: 140219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.047463] LustreError: 140219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2351.576302] Lustre: server umount lustre-OST0000 complete [ 2354.148525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.163672] Lustre: Skipped 2 previous similar messages [ 2360.105859] Lustre: server umount lustre-MDT0000 complete [ 2362.555816] LustreError: 138380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910942 with bad export cookie 10736460435219199424 [ 2362.561822] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.565845] LustreError: 138380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2362.788384] Lustre: server umount lustre-MDT0001 complete [ 2374.885890] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2376.693921] Key type lgssc unregistered [ 2376.904838] LNet: 144768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.910977] LNetError: 144768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.924615] LNet: Removed LNI 192.168.203.134@tcp [ 2377.513473] Key type .llcrypt unregistered [ 2377.515738] Key type ._llcrypt unregistered [ 2400.909245] Key type ._llcrypt registered [ 2400.912738] Key type .llcrypt registered [ 2400.979223] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2409.355412] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2409.776699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.854168] alg: No test for adler32 (adler32-zlib) [ 2410.783228] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2410.925491] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2412.521081] Key type lgssc registered [ 2413.075905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2431.444388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2431.456417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.558325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2432.596381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2432.638288] Lustre: lustre-MDT0000: new disk, initializing [ 2432.681505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.692443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2434.464601] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2439.500646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.554828] Lustre: 148896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2439.588607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2439.591558] Lustre: Skipped 1 previous similar message [ 2439.629329] Lustre: lustre-MDT0001: new disk, initializing [ 2439.667325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.684066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2439.688752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2441.309813] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2445.853285] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2448.214670] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2451.460471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.605948] Lustre: lustre-OST0000: new disk, initializing [ 2451.608883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2451.614104] Lustre: 150863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2451.659722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.572173] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2458.112445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2458.116721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2458.161124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2459.181928] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2461.431272] Lustre: Failing over lustre-OST0000 [ 2461.532385] Lustre: server umount lustre-OST0000 complete [ 2475.385906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.520827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.531223] Lustre: Skipped 1 previous similar message [ 2475.538772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.550967] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2475.845373] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2478.341892] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2478.560085] Lustre: 147121:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788911042/real 1788911042] req@00000000e29a126b x1875809130278016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788911058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2480.054199] Lustre: Failing over lustre-MDT0000 [ 2480.165627] Lustre: server umount lustre-MDT0000 complete [ 2480.610836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.612103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.615925] Lustre: Skipped 1 previous similar message [ 2483.680140] Lustre: 147123:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788911047/real 1788911047] req@00000000a8913aee x1875809130279168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788911063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2483.701080] Lustre: 147123:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2485.732294] LustreError: 148902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.742231] LustreError: 148902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2489.888136] Lustre: 147123:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788911053/real 1788911053] req@00000000469795fa x1875809130280320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788911069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2490.851646] LustreError: 150379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.859027] LustreError: 150379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2493.971563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.041739] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.048474] Lustre: 148898:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2494.144775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2494.166531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.214325] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2496.034660] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2496.305211] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2499.558807] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2499.567190] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2499.589304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2504.674852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.680816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.685686] Lustre: Skipped 2 previous similar messages [ 2504.686221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.690272] Lustre: Skipped 1 previous similar message [ 2506.980343] Lustre: server umount lustre-OST0000 complete [ 2509.794348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2509.794809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.805134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2509.807743] Lustre: Skipped 1 previous similar message [ 2514.647785] Lustre: server umount lustre-MDT0000 complete [ 2514.913995] LustreError: 148907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.920620] LustreError: 148907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2516.123582] LustreError: 148889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911095 with bad export cookie 7067574709148435727 [ 2516.125629] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.129087] LustreError: 148889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2516.263555] Lustre: server umount lustre-MDT0001 complete [ 2520.762821] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2522.089646] Key type lgssc unregistered [ 2522.256593] LNet: 154505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.259557] LNetError: 154505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.268465] LNet: Removed LNI 192.168.203.134@tcp [ 2522.644102] Key type .llcrypt unregistered [ 2522.645743] Key type ._llcrypt unregistered [ 2544.846129] Key type ._llcrypt registered [ 2544.847567] Key type .llcrypt registered [ 2544.899272] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2545.320052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.387258] alg: No test for adler32 (adler32-zlib) [ 2546.297266] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2546.423632] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2548.024423] Key type lgssc registered [ 2548.581494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.402174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2552.414250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.645664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.528796] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2560.097197] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2562.458232] LustreError: 156047:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2562.463145] LustreError: 156047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2562.467779] LustreError: 156047:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2562.630195] Lustre: server umount lustre-MDT0000 complete [ 2568.893462] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2572.628569] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2587.262027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.371726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2587.387070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2587.425639] Lustre: lustre-MDT0000: new disk, initializing [ 2587.459160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.467475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2589.190438] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2594.406748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.468145] Lustre: 159345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2594.502862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2594.508067] Lustre: Skipped 1 previous similar message [ 2594.556631] Lustre: lustre-MDT0001: new disk, initializing [ 2594.610437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2594.635957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2594.641599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2596.369605] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2600.800259] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2603.107959] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2605.957680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.069711] Lustre: lustre-OST0000: new disk, initializing [ 2606.073750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2606.078747] Lustre: 161310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2606.117509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.494139] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2612.877894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2613.246862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2613.251418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2613.297112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2619.844372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2620.458415] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:46:39 (1788911199) [ 2690.016481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.016704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.019923] Lustre: Skipped 2 previous similar messages [ 2690.022457] Lustre: Skipped 2 previous similar messages [ 2695.136551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.139704] Lustre: Skipped 2 previous similar messages [ 2700.256653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.258677] Lustre: Skipped 2 previous similar messages [ 2703.328115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2703.398491] Lustre: server umount lustre-MDT0000 complete [ 2704.595410] LustreError: 161309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911284 with bad export cookie 6018505927383266304 [ 2704.596378] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.598598] LustreError: 161309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2705.376477] LustreError: 162028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.376570] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.381036] LustreError: 162028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2705.386948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2709.408443] LustreError: 159350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.408451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2710.749934] Lustre: server umount lustre-MDT0001 complete [ 2726.368117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.414071] Lustre: server umount lustre-OST0000 complete [ 2729.314183] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2732.135490] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2742.872891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.950274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2742.961050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2742.994809] Lustre: lustre-MDT0000: new disk, initializing [ 2743.013544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.019640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2744.238874] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2748.143985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.183745] Lustre: 165942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2748.205599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2748.207600] Lustre: Skipped 1 previous similar message [ 2748.242467] Lustre: lustre-MDT0001: new disk, initializing [ 2748.278204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2748.282809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2749.523641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2752.953845] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.587664] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2756.658566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.752829] Lustre: lustre-OST0000: new disk, initializing [ 2756.754882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2756.756987] Lustre: 167906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2756.786181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.789239] Lustre: Skipped 1 previous similar message [ 2758.543307] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2759.988020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2759.990916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2760.015380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2762.003768] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.322593] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2763.396848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2764.677649] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2764.744057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2770.401806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.402556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.404342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.404630] LustreError: Skipped 1 previous similar message [ 2770.408087] Lustre: Skipped 1 previous similar message [ 2771.406735] Lustre: server umount lustre-OST0000 complete [ 2773.984766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2773.988315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.773193] Lustre: server umount lustre-MDT0000 complete [ 2779.962928] LustreError: 165933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911359 with bad export cookie 6018505927383425120 [ 2779.964806] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.967171] LustreError: 165933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2780.073627] Lustre: server umount lustre-MDT0001 complete [ 2782.738843] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:49:22 (1788911362) [ 2787.772272] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2790.603249] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2802.011987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.094695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2802.108365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2802.148497] Lustre: lustre-MDT0000: new disk, initializing [ 2802.167971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.173658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2803.526714] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2807.586285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.627822] Lustre: 172654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2807.722797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2809.025882] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2812.942419] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2814.852265] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2817.010991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.116571] Lustre: 174627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2819.050203] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2822.825182] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2824.188978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2824.231995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2834.403482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.403502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.407751] Lustre: Skipped 1 previous similar message [ 2834.409911] LustreError: Skipped 1 previous similar message [ 2834.410521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2834.416072] Lustre: Skipped 5 previous similar messages [ 2844.128101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2844.192439] Lustre: server umount lustre-OST0000 complete [ 2848.737195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2848.740102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.672940] Lustre: server umount lustre-MDT0000 complete [ 2852.860309] LustreError: 173453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911432 with bad export cookie 6018505927383425988 [ 2852.862794] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.865709] LustreError: 173453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2852.966106] Lustre: server umount lustre-MDT0001 complete [ 2855.713416] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:50:35 (1788911435) [ 2860.579714] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2863.285055] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2873.904573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.977041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2873.986581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2873.988480] Lustre: Skipped 3 previous similar messages [ 2874.020620] Lustre: lustre-MDT0000: new disk, initializing [ 2874.021985] Lustre: Skipped 2 previous similar messages [ 2874.040262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.041995] Lustre: Skipped 2 previous similar messages [ 2874.046621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2874.049408] Lustre: Skipped 2 previous similar messages [ 2875.307672] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2879.310851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.352629] Lustre: 178942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2879.454542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2880.772532] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2884.329352] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2886.039895] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2888.236937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.345034] Lustre: 180916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2889.826131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2890.303841] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2894.007950] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2899.939726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2899.939842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.942706] LustreError: Skipped 1 previous similar message [ 2899.946105] Lustre: Skipped 1 previous similar message [ 2899.946854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.950861] Lustre: Skipped 6 previous similar messages [ 2901.970719] Lustre: server umount lustre-OST0000 complete [ 2909.342930] Lustre: server umount lustre-MDT0000 complete [ 2910.177197] LustreError: 178977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.183458] LustreError: 178977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2910.713736] LustreError: 180915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911490 with bad export cookie 6018505927383427059 [ 2910.718520] LustreError: 180915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.718662] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.841479] Lustre: server umount lustre-MDT0001 complete [ 2914.595399] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2915.698125] Key type lgssc unregistered [ 2915.828658] LNet: 182766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.834075] LNetError: 182766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.844842] LNet: Removed LNI 192.168.203.134@tcp [ 2916.171283] Key type .llcrypt unregistered [ 2916.172588] Key type ._llcrypt unregistered [ 2925.211277] Key type ._llcrypt registered [ 2925.212620] Key type .llcrypt registered [ 2925.255703] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2925.594525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2925.651088] alg: No test for adler32 (adler32-zlib) [ 2926.500341] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2926.602188] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2928.184148] Key type lgssc registered [ 2928.534977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.994533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2931.000200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.137518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.430717] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2936.058350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.182146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.522582] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2939.835463] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2941.474427] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2943.541078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.657941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.482561] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2947.702723] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.992647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2971.616695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.618995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.622973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.665262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.665750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.666917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.673437] Lustre: Skipped 1 previous similar message [ 2975.697967] Lustre: server umount lustre-OST0000 complete [ 2977.249024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2977.251377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.257163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.131091] Lustre: server umount lustre-MDT0000 complete [ 2983.904963] LustreError: 184156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.910374] LustreError: 184156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2984.299181] LustreError: 184137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911563 with bad export cookie 6171026393367914233 [ 2984.302913] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.303916] LustreError: 184137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2984.434282] Lustre: server umount lustre-MDT0001 complete [ 2987.793935] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2988.765970] Key type lgssc unregistered [ 2988.879529] LNet: 187708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.881488] LNetError: 187708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.889233] LNet: Removed LNI 192.168.203.134@tcp [ 2989.181841] Key type .llcrypt unregistered [ 2989.182819] Key type ._llcrypt unregistered [ 2997.752553] Key type ._llcrypt registered [ 2997.753636] Key type .llcrypt registered [ 2997.790751] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2998.062351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.070667] alg: No test for adler32 (adler32-zlib) [ 2998.929853] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 2999.031043] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3000.616151] Key type lgssc registered [ 3000.998798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.650759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3003.659129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.801531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.132648] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3008.888408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.011681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3010.314817] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3012.611987] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3014.334802] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3016.457416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.574199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.436553] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3020.745202] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3029.926368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3034.080789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.082926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.088088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.105902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.107207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.108023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.471314] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 3036.474419] Lustre: Skipped 1 previous similar message [ 3039.564700] Lustre: server umount lustre-OST0000 complete [ 3041.816891] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8bf0458fa800.ost_server_uuid 50 [ 3042.326163] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3052.230180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.231507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.234338] Lustre: Skipped 1 previous similar message [ 3056.288928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3056.290850] Lustre: Skipped 2 previous similar messages [ 3058.383203] Lustre: server umount lustre-MDT0000 complete [ 3059.536305] LustreError: 189091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911639 with bad export cookie 10374194995711481449 [ 3059.538270] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.540387] LustreError: 189091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3059.642824] Lustre: server umount lustre-MDT0001 complete [ 3062.258769] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:54:01 (1788911641) [ 3064.359329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.510390] LustreError: 193020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.516743] LustreError: 193020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3064.534125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.802725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3068.597362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.718411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.016033] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3072.339802] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3074.075180] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3076.193565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.001916] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3080.131338] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3082.247053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.317180] Lustre: lustre-OST0001: new disk, initializing [ 3082.318753] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3082.320985] Lustre: 195859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3082.348328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.350993] Lustre: Skipped 1 previous similar message [ 3084.096779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3084.100224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3084.127178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3084.172888] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3087.714536] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.381710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.387502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3094.584809] Lustre: server umount lustre-OST0000 complete [ 3095.521934] LustreError: 195424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.527310] LustreError: 195424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3096.423966] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3096.491299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3097.806683] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3097.877666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3098.417357] LustreError: 196668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.642784] LustreError: 194840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.647748] LustreError: 194840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3105.762943] LustreError: 194843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.767241] LustreError: 194843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3109.857540] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3109.858129] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.859871] LustreError: Skipped 1 previous similar message [ 3109.864414] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3109.867297] Lustre: Skipped 4 previous similar messages [ 3113.865890] Lustre: server umount lustre-OST0001 complete [ 3121.238169] Lustre: server umount lustre-MDT0000 complete [ 3122.383074] LustreError: 193045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911701 with bad export cookie 10374194995711482513 [ 3122.385859] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.386333] LustreError: 193045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3122.486135] Lustre: server umount lustre-MDT0001 complete [ 3129.536648] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:55:08 (1788911708) [ 3131.465165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.546036] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3131.555240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3131.615721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.778263] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3135.382956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.412517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3135.424236] Lustre: 199273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3136.779501] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3139.104673] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3140.797247] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3142.892575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.955315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3144.736224] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3146.993939] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3147.081601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3149.135766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.159668] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3151.088694] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3152.217178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3153.354624] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3157.475808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3157.476381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.478089] LustreError: Skipped 1 previous similar message [ 3157.478654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3157.482013] Lustre: Skipped 4 previous similar messages [ 3157.483217] Lustre: Skipped 8 previous similar messages [ 3160.249222] Lustre: server umount lustre-OST0000 complete [ 3162.596996] LustreError: 201121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.601391] LustreError: 201121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3172.835465] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.838389] LustreError: Skipped 1 previous similar message [ 3176.783531] Lustre: server umount lustre-OST0001 complete [ 3177.953176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.958114] Lustre: Skipped 3 previous similar messages [ 3192.288121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3192.348272] Lustre: server umount lustre-MDT0000 complete [ 3193.313036] LustreError: 199283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.318778] LustreError: 199283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3193.473286] LustreError: 200018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911773 with bad export cookie 10374194995711483675 [ 3193.475701] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.477612] LustreError: 200018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.583815] Lustre: server umount lustre-MDT0001 complete [ 3200.925631] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:56:20 (1788911780) [ 3205.767845] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3208.683375] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3219.563848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.640297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3219.651922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3219.686420] Lustre: lustre-MDT0000: new disk, initializing [ 3219.705570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.708396] Lustre: Skipped 3 previous similar messages [ 3219.714323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3220.939487] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3224.875562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.918377] Lustre: 207350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3224.940687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3224.942732] Lustre: Skipped 1 previous similar message [ 3224.979095] Lustre: lustre-MDT0001: new disk, initializing [ 3225.019898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3225.023170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3226.307081] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3229.822424] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3231.547501] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3233.723849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3233.827405] Lustre: lustre-OST0000: new disk, initializing [ 3233.829876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3233.833581] Lustre: 209315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3235.672616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3237.054117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3237.057902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3237.096307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3239.326785] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3240.702198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3240.772453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3242.105477] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3242.176071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3247.586405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3247.587060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.589543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.593516] Lustre: Skipped 1 previous similar message [ 3247.597124] Lustre: Skipped 11 previous similar messages [ 3257.312105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3257.363207] Lustre: server umount lustre-OST0000 complete [ 3264.661539] Lustre: server umount lustre-MDT0000 complete [ 3265.799992] LustreError: 207341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911845 with bad export cookie 10374194995711485439 [ 3265.802660] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.804267] LustreError: 207341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3265.912530] Lustre: server umount lustre-MDT0001 complete [ 3268.547378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.689659] LustreError: 211530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.895317] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3272.482144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.860550] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3276.065811] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3277.803416] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3280.016372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.040934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3284.337447] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3285.693120] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3293.886711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3295.135773] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3295.196850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3298.785804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.788940] LustreError: Skipped 2 previous similar messages [ 3301.840348] Lustre: server umount lustre-OST0000 complete [ 3303.666503] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3303.732117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3304.992414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3328.591553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.732629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.735479] Lustre: Skipped 5 previous similar messages [ 3330.697576] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3333.203690] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.742478] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3336.842174] Lustre: server umount lustre-OST0000 complete [ 3338.088788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.092918] Lustre: Skipped 5 previous similar messages [ 3340.180849] Lustre: server umount lustre-MDT0000 complete [ 3341.414273] LustreError: 211511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911920 with bad export cookie 10374194995711486307 [ 3341.417406] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.417753] LustreError: 211511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3341.535281] Lustre: server umount lustre-MDT0001 complete [ 3344.316362] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:58:43 (1788911923) [ 3346.418458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.571472] LustreError: 217560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.993619] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3350.891336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.477677] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3354.997972] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3356.791413] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3359.087876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.092802] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3363.279577] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3364.564543] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3371.740922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3372.987796] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3373.047436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.107976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.172410] Lustre: lustre-OST0001: new disk, initializing [ 3375.173805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3375.175970] Lustre: 220978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3377.013925] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3380.449973] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3381.250374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3381.253111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3381.289219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3381.748883] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3381.814923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3383.123222] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3383.198251] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3386.340504] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.341359] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3386.342983] LustreError: Skipped 2 previous similar messages [ 3386.344619] Lustre: Skipped 11 previous similar messages [ 3389.885770] Lustre: server umount lustre-OST0001 complete [ 3391.804197] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3391.872258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3393.162340] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3393.230111] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3416.609187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.484854] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3420.705940] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3421.173833] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3427.833210] Lustre: server umount lustre-OST0001 complete [ 3445.834619] Lustre: server umount lustre-OST0000 complete [ 3453.205381] Lustre: server umount lustre-MDT0000 complete [ 3454.379046] LustreError: 217542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912033 with bad export cookie 10374194995711487238 [ 3454.382420] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.382698] LustreError: 217542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3460.529871] Lustre: server umount lustre-MDT0001 complete [ 3467.960819] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:00:47 (1788912047) [ 3469.963634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.052323] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3470.063268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3470.127505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.129232] Lustre: Skipped 5 previous similar messages [ 3471.320732] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3474.043083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.076507] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3474.085202] Lustre: 226205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3475.466918] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3477.783921] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3479.450032] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.481497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.547445] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3483.395653] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3485.551478] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3488.973609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.817184] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3493.013856] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.500467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3496.684667] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3496.747349] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3497.956519] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3498.020254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3499.610165] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bf048f79800.ost_server_uuid 50 [ 3500.048254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bf048f79800.ost_server_uuid in FULL state after 0 sec [ 3500.338700] Lustre: Permanently deactivating lustre-OST0001 [ 3500.342969] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3501.028214] Lustre: Permanently reactivating lustre-OST0001 [ 3502.052901] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.052994] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3502.056458] Lustre: Skipped 7 previous similar messages [ 3502.058139] LustreError: Skipped 1 previous similar message [ 3507.964929] Lustre: server umount lustre-OST0001 complete [ 3512.292106] LustreError: 228048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.297737] LustreError: 228048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 3515.340894] Lustre: server umount lustre-OST0000 complete [ 3522.717119] Lustre: server umount lustre-MDT0000 complete [ 3523.868404] LustreError: 226196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912103 with bad export cookie 10374194995711488498 [ 3523.872353] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.872577] LustreError: 226196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3523.993572] Lustre: server umount lustre-MDT0001 complete [ 3530.878078] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:01:50 (1788912110) [ 3533.147130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.238829] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3533.251174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3533.254586] Lustre: Skipped 5 previous similar messages [ 3534.577396] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3537.193820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.228476] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3537.230340] Lustre: Skipped 1 previous similar message [ 3537.240086] Lustre: 232902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3538.629965] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3540.911134] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3542.635285] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.764799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.836831] Lustre: Permanently deactivating lustre-OST0000 [ 3544.840888] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3546.678550] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3548.982178] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3551.155073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.281684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3551.332965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3553.182078] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3555.465905] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3555.985621] Lustre: Permanently reactivating lustre-OST0000 [ 3561.462191] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3561.464885] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3561.468745] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3561.492955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3561.502469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3572.987173] Lustre: server umount lustre-OST0001 complete [ 3580.238336] Lustre: server umount lustre-OST0000 complete [ 3581.659612] Lustre: server umount lustre-MDT0000 complete [ 3582.904029] LustreError: 233642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912162 with bad export cookie 10374194995711490157 [ 3582.904849] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3582.908116] LustreError: 233642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.107853] Lustre: server umount lustre-MDT0001 complete [ 3592.867758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3593.932106] Key type lgssc unregistered [ 3594.060461] LNet: 237771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3594.063602] LNetError: 237771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3594.071459] LNet: Removed LNI 192.168.203.134@tcp [ 3594.356438] Key type .llcrypt unregistered [ 3594.358268] Key type ._llcrypt unregistered [ 3602.512050] Key type ._llcrypt registered [ 3602.513756] Key type .llcrypt registered [ 3602.559547] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3602.897299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.935566] alg: No test for adler32 (adler32-zlib) [ 3603.780495] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 3603.871658] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3605.448112] Key type lgssc registered [ 3605.782333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3608.915238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3608.922505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.062721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.399658] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3614.164720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.206166] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3614.215062] Lustre: 239199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3614.219195] Lustre: Permanently deactivating lustre-MDT0001 [ 3614.224397] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3614.344638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.620063] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3617.917556] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.593054] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3619.822389] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3621.640542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.734664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.489737] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3623.778960] LustreError: 241048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.782648] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3623.783954] LustreError: 241048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3624.802675] LustreError: 241049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.790151] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3627.930607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.995529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3629.774910] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3632.009721] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3633.126663] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3635.176566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3635.176916] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3637.946920] Lustre: Permanently reactivating lustre-MDT0001 [ 3637.953672] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3637.955778] Lustre: Skipped 2 previous similar messages [ 3639.414729] Lustre: lustre-MDT0001: Client 935a176e-58fa-4459-8a55-445246623bd5 (at 192.168.203.34@tcp) reconnecting [ 3641.815894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3642.362023] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3643.387705] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.391484] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3643.395237] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3643.400928] LustreError: 243125:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3643.401557] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3643.405334] LustreError: 243125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3643.412373] LustreError: 243125:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -108 [ 3643.415480] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3643.415480] [ 3643.418113] LustreError: 239201:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3644.035814] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3644.098292] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3645.526105] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3645.608147] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3646.410775] Lustre: Permanently deactivating lustre-MDT0001 [ 3646.415250] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3646.418383] Lustre: Skipped 2 previous similar messages [ 3651.068395] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3657.698639] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3657.701249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.705404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.769219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.771222] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.775849] Lustre: Skipped 2 previous similar messages [ 3662.235493] Lustre: server umount lustre-MDT0000 complete [ 3663.447152] LustreError: 239948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912243 with bad export cookie 829884404427249658 [ 3663.450622] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.451734] LustreError: 239948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3663.579926] Lustre: server umount lustre-MDT0001 complete [ 3670.904416] Lustre: server umount lustre-OST0000 complete [ 3674.680554] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:04:14 (1788912254) [ 3678.251508] Lustre: server umount lustre-OST0001 complete [ 3680.146913] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3683.324848] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3694.363410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.458834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3694.461929] Lustre: Skipped 2 previous similar messages [ 3694.470401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3694.510781] Lustre: lustre-MDT0000: new disk, initializing [ 3694.534222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.540747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3695.891298] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3699.911804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.949364] Lustre: 247499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3699.974222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3699.975806] Lustre: Skipped 1 previous similar message [ 3700.014320] Lustre: lustre-MDT0001: new disk, initializing [ 3700.057527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3700.061035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3701.393243] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3705.065174] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3706.901711] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3709.266244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.381337] Lustre: lustre-OST0000: new disk, initializing [ 3709.382934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3709.385620] Lustre: 249468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3709.414979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.417672] Lustre: Skipped 1 previous similar message [ 3711.359085] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3711.420751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3711.426378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3711.472427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3715.092969] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3715.746105] LustreError: 247535:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3719.995877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.068412] Lustre: lustre-OST0001: new disk, initializing [ 3720.070954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3720.074536] Lustre: 250584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3722.126651] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3725.840061] LustreError: 247535:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3725.847294] LustreError: 247506:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3725.946269] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3726.482822] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3727.361919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3727.367001] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3727.392389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3735.936131] LustreError: 247506:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3737.570861] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3737.571048] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.573147] LustreError: Skipped 1 previous similar message [ 3737.574088] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3737.574094] Lustre: Skipped 3 previous similar messages [ 3737.579098] Lustre: Skipped 1 previous similar message [ 3742.657328] Lustre: server umount lustre-OST0001 complete [ 3742.692760] LustreError: 249453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.697231] LustreError: 249453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3747.811989] LustreError: 249453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3747.817381] LustreError: 249453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3752.933528] LustreError: 250189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.938396] LustreError: 250189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3757.537184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.537264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.540557] LustreError: Skipped 1 previous similar message [ 3757.544430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.547845] Lustre: Skipped 3 previous similar messages [ 3760.787134] Lustre: server umount lustre-OST0000 complete [ 3763.170974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.174362] Lustre: Skipped 3 previous similar messages [ 3768.277546] Lustre: server umount lustre-MDT0000 complete [ 3768.289026] LustreError: 247511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.296597] LustreError: 247511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3769.474460] LustreError: 247492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912349 with bad export cookie 829884404427252472 [ 3769.478027] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3769.479727] LustreError: 247492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3775.667036] Lustre: server umount lustre-MDT0001 complete [ 3779.157538] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3780.174169] Key type lgssc unregistered [ 3780.296382] LNet: 252613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3780.298794] LNetError: 252613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3780.309325] LNet: Removed LNI 192.168.203.134@tcp [ 3780.638079] Key type .llcrypt unregistered [ 3780.639917] Key type ._llcrypt unregistered [ 3794.473729] Key type ._llcrypt registered [ 3794.474784] Key type .llcrypt registered [ 3794.523805] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3794.819125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3794.836586] alg: No test for adler32 (adler32-zlib) [ 3795.683401] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 3795.791174] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3797.384166] Key type lgssc registered [ 3797.789170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.348601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3800.356143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.446187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3801.455382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.511473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.857649] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3805.676096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.710089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3805.722326] Lustre: 254791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3805.813897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3807.131739] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3809.479585] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3811.236226] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3813.426118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.490300] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3813.556048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.795934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3815.389431] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3817.701619] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3859.939443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.941573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3859.941582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3859.943116] Lustre: Skipped 1 previous similar message [ 3863.008630] Lustre: server umount lustre-OST0000 complete [ 3864.442162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3868.645299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3868.766842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.753785] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3873.189189] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3880.040821] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3880.047514] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3880.047608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3911.139679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.140366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.141016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3911.141021] Lustre: Skipped 1 previous similar message [ 3911.144091] Lustre: Skipped 1 previous similar message [ 3911.152848] LustreError: Skipped 2 previous similar messages [ 3912.529779] Lustre: server umount lustre-OST0000 complete [ 3913.696786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3913.700341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.703868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.705362] Lustre: Skipped 1 previous similar message [ 3916.257120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3916.257566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.764088] Lustre: server umount lustre-MDT0000 complete [ 3920.900655] LustreError: 254782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912500 with bad export cookie 13452036254465149161 [ 3920.903305] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.904958] LustreError: 254782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3921.011494] Lustre: server umount lustre-MDT0001 complete [ 3924.342734] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3925.320558] Key type lgssc unregistered [ 3925.450307] LNet: 260200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3925.452506] LNetError: 260200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3925.459303] LNet: Removed LNI 192.168.203.134@tcp [ 3925.750665] Key type .llcrypt unregistered [ 3925.751625] Key type ._llcrypt unregistered [ 3933.935803] Key type ._llcrypt registered [ 3933.936790] Key type .llcrypt registered [ 3933.974705] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3934.268647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3934.308615] alg: No test for adler32 (adler32-zlib) [ 3935.152935] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 3935.237939] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3936.824187] Key type lgssc registered [ 3937.177048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3939.650197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3939.655732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.789279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.018144] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3944.647968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.754318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3945.992640] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3948.211107] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3949.977252] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3952.224750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3952.340124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.295877] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3956.660539] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3965.670636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3976.162508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.163018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.164671] LustreError: Skipped 1 previous similar message [ 3976.170038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.172095] Lustre: Skipped 1 previous similar message [ 3980.628226] Lustre: server umount lustre-OST0000 complete [ 3985.888809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3985.892048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.897296] Lustre: Skipped 1 previous similar message [ 3985.899554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.057283] Lustre: server umount lustre-MDT0000 complete [ 3989.267261] LustreError: 262324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912568 with bad export cookie 12754363585901501691 [ 3989.270549] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3989.270963] LustreError: 262324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3989.389253] Lustre: server umount lustre-MDT0001 complete [ 3992.903163] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3993.999372] Key type lgssc unregistered [ 3994.134800] LNet: 265908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3994.139282] LNetError: 265908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3994.152349] LNet: Removed LNI 192.168.203.134@tcp [ 3994.485335] Key type .llcrypt unregistered [ 3994.486756] Key type ._llcrypt unregistered [ 4001.572437] Key type ._llcrypt registered [ 4001.573674] Key type .llcrypt registered [ 4001.622810] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4002.016071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4002.047146] alg: No test for adler32 (adler32-zlib) [ 4002.902693] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4002.995758] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4004.576161] Key type lgssc registered [ 4004.953484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4007.516843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4007.522936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.670351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4010.135283] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4013.100526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.233834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.640134] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4017.178128] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.037358] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4021.305078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.423564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.390513] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4025.918367] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4034.727413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4040.163387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.164972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.165189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.169095] Lustre: Skipped 1 previous similar message [ 4045.281450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.283208] Lustre: Skipped 2 previous similar messages [ 4046.031436] Lustre: server umount lustre-OST0000 complete [ 4049.377139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4049.381427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.387510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.402404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.467405] Lustre: server umount lustre-MDT0000 complete [ 4054.587902] LustreError: 267129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912634 with bad export cookie 13449764604696859991 [ 4054.590285] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.591107] LustreError: 267129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4054.690113] Lustre: server umount lustre-MDT0001 complete [ 4058.114495] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4059.034702] Key type lgssc unregistered [ 4059.162372] LNet: 270798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.164803] LNetError: 270798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.177257] LNet: Removed LNI 192.168.203.134@tcp [ 4059.455352] Key type .llcrypt unregistered [ 4059.456377] Key type ._llcrypt unregistered [ 4068.159185] Key type ._llcrypt registered [ 4068.160377] Key type .llcrypt registered [ 4068.206211] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4068.597748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.645317] alg: No test for adler32 (adler32-zlib) [ 4069.494777] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4069.578302] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4071.152112] Key type lgssc registered [ 4071.464540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4073.835932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4073.842148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.966029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.359571] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4079.235146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.358767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4080.722664] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4083.155588] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4085.069301] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4087.473203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.614589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.657327] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4092.076392] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.940268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4110.306470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4110.308984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.310580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.313969] Lustre: Skipped 1 previous similar message [ 4110.316903] Lustre: Skipped 1 previous similar message [ 4115.426031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.427886] Lustre: Skipped 1 previous similar message [ 4115.857752] Lustre: server umount lustre-OST0000 complete [ 4120.546206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.546369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.552973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.554562] Lustre: Skipped 1 previous similar message [ 4123.224574] Lustre: server umount lustre-MDT0000 complete [ 4124.426725] LustreError: 272168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912704 with bad export cookie 11305435526163964092 [ 4124.430160] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4124.433341] LustreError: 272168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4124.547037] Lustre: server umount lustre-MDT0001 complete [ 4128.024421] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4129.060571] Key type lgssc unregistered [ 4129.188428] LNet: 276507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4129.193445] LNetError: 276507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.203309] LNet: Removed LNI 192.168.203.134@tcp [ 4129.491382] Key type .llcrypt unregistered [ 4129.492504] Key type ._llcrypt unregistered [ 4136.333698] Key type ._llcrypt registered [ 4136.334720] Key type .llcrypt registered [ 4136.373056] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4136.786561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4136.795868] alg: No test for adler32 (adler32-zlib) [ 4137.658133] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4137.750838] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4139.328169] Key type lgssc registered [ 4139.676982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4142.178830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4142.186492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.339823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.607565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4147.354724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.474646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4148.811539] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4151.167357] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4152.964211] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4155.173710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4155.279674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.133133] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4159.444170] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4167.592267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4173.792828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4173.796287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.802661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.891649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.892364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.892455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.894627] Lustre: Skipped 1 previous similar message [ 4179.538895] Lustre: server umount lustre-OST0000 complete [ 4183.009620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.015055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.017154] Lustre: Skipped 1 previous similar message [ 4188.129109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.131959] Lustre: Skipped 1 previous similar message [ 4193.248966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.250731] Lustre: Skipped 1 previous similar message [ 4195.296121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4195.369746] Lustre: server umount lustre-MDT0000 complete [ 4196.655311] LustreError: 277681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912776 with bad export cookie 3903868319369120583 [ 4196.659135] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.775951] Lustre: server umount lustre-MDT0001 complete [ 4200.224125] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4201.192234] Key type lgssc unregistered [ 4201.326388] LNet: 281432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.328703] LNetError: 281432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.335284] LNet: Removed LNI 192.168.203.134@tcp [ 4201.631876] Key type .llcrypt unregistered [ 4201.632953] Key type ._llcrypt unregistered [ 4205.094973] Key type ._llcrypt registered [ 4205.096433] Key type .llcrypt registered [ 4205.132830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4210.374594] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4216.352227] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4216.725139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4216.747580] alg: No test for adler32 (adler32-zlib) [ 4217.586180] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4217.660497] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4219.240101] Key type lgssc registered [ 4219.627088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4235.809785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4235.821623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.975398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4236.999564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4237.046727] Lustre: lustre-MDT0000: new disk, initializing [ 4237.097329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.108259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.122281] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4248.050932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.129165] Lustre: 284953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4248.164759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4248.169035] Lustre: Skipped 1 previous similar message [ 4248.224611] Lustre: lustre-MDT0001: new disk, initializing [ 4248.279111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4248.300096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4248.306139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4250.207328] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4254.225708] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4256.004062] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4258.251170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.357343] Lustre: lustre-OST0000: new disk, initializing [ 4258.358870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4258.361213] Lustre: 286920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4258.392331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.310499] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4263.939492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4263.944650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4263.996351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4264.191777] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4266.068418] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4266.170877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4268.126576] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4268.235214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.147733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.148746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.150415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.161483] Lustre: Skipped 1 previous similar message [ 4279.267267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.270520] Lustre: Skipped 2 previous similar messages [ 4283.360117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4283.414060] Lustre: server umount lustre-OST0000 complete [ 4289.505673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.505808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.506616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.510860] Lustre: Skipped 1 previous similar message [ 4289.513817] LustreError: Skipped 1 previous similar message [ 4294.627235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4294.630120] Lustre: Skipped 2 previous similar messages [ 4299.232384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4299.298059] Lustre: server umount lustre-MDT0000 complete [ 4300.604070] LustreError: 285756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912880 with bad export cookie 12121993158115692187 [ 4300.604901] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.608741] LustreError: 285756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4300.724610] Lustre: server umount lustre-MDT0001 complete [ 4303.951935] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:14:43 (1788912883) [ 4306.343894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.527987] LustreError: 289332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.534961] LustreError: 289332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4306.551407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.942371] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4311.003846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.186628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.680930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4315.289418] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4317.320936] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4319.899479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.038743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.121364] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4324.605224] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.657348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.660096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.660906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.666432] Lustre: Skipped 1 previous similar message [ 4338.671013] Lustre: Skipped 1 previous similar message [ 4344.014629] Lustre: server umount lustre-OST0000 complete [ 4347.361526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4347.364792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.372114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.374766] Lustre: Skipped 2 previous similar messages [ 4351.589818] Lustre: server umount lustre-MDT0000 complete [ 4353.024343] LustreError: 289314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912932 with bad export cookie 12121993158115693069 [ 4353.028067] LustreError: 290069:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4353.029490] LustreError: 289314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4353.029612] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.156507] Lustre: server umount lustre-MDT0001 complete [ 4357.049201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4358.184548] Key type lgssc unregistered [ 4358.321511] LNet: 292883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4358.324989] LNetError: 292883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4358.332836] LNet: Removed LNI 192.168.203.134@tcp [ 4358.662533] Key type .llcrypt unregistered [ 4358.663996] Key type ._llcrypt unregistered [ 4367.932859] Key type ._llcrypt registered [ 4367.934463] Key type .llcrypt registered [ 4367.975887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4374.386514] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4374.711977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4374.719874] alg: No test for adler32 (adler32-zlib) [ 4375.578405] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4375.666983] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4377.248181] Key type lgssc registered [ 4377.638285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4391.911562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4391.918459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.990718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4393.000265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4393.037305] Lustre: lustre-MDT0000: new disk, initializing [ 4393.059119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.068052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4394.282590] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4398.220151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.267515] Lustre: 296677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4398.288768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4398.291085] Lustre: Skipped 1 previous similar message [ 4398.324236] Lustre: lustre-MDT0001: new disk, initializing [ 4398.356051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4398.367890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4398.371054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4399.645981] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4403.176233] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.922178] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4406.994174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.092102] Lustre: lustre-OST2710: new disk, initializing [ 4407.094270] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4407.097399] Lustre: 298643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4407.125063] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4408.969181] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4410.616444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4410.619736] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4410.659568] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4412.673398] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4414.901014] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.971556] Lustre: lustre-OST03e8: new disk, initializing [ 4414.973838] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4414.976621] Lustre: 299705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4415.017358] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4416.835519] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4418.372091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4418.376018] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4418.401859] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4419.264751] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4421.888923] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4421.954436] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4423.280520] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4423.358377] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4424.691649] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4424.766182] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4426.064168] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4426.136476] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4428.774173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.776180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.779204] Lustre: Skipped 3 previous similar messages [ 4428.780862] Lustre: Skipped 2 previous similar messages [ 4433.264692] Lustre: server umount lustre-MDT0000 complete [ 4434.652469] LustreError: 296666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913014 with bad export cookie 9938142055457982804 [ 4434.655724] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.657448] LustreError: 296666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4434.782455] Lustre: server umount lustre-MDT0001 complete [ 4446.221606] Lustre: server umount lustre-OST2710 complete [ 4457.814690] Lustre: server umount lustre-OST03e8 complete [ 4464.052884] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4469.462979] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4501.826650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.033703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4502.058158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4502.112336] Lustre: lustre-MDT0000: new disk, initializing [ 4502.202371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.219226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4506.885914] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4516.301977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.397808] Lustre: 304950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4516.435200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4516.438527] Lustre: Skipped 1 previous similar message [ 4516.483712] Lustre: lustre-MDT0001: new disk, initializing [ 4516.533811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4516.566039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4516.574950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4519.158646] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4524.105316] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4526.668216] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4529.581729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4529.707303] Lustre: 306915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4532.290446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4536.888606] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.345117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4537.350326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4537.387904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.606300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4538.699863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4540.388054] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4540.479786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4542.436598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4542.441846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.443453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4542.453389] Lustre: Skipped 1 previous similar message [ 4542.456027] Lustre: Skipped 1 previous similar message [ 4543.262562] Lustre: server umount lustre-OST0000 complete [ 4547.554055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.556285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4547.556460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.556463] Lustre: Skipped 1 previous similar message [ 4547.566415] Lustre: Skipped 1 previous similar message [ 4547.569981] LustreError: Skipped 1 previous similar message [ 4552.672935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.676700] Lustre: Skipped 2 previous similar messages [ 4557.793079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.796111] Lustre: Skipped 1 previous similar message [ 4559.328715] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4559.408374] Lustre: server umount lustre-MDT0000 complete [ 4560.995620] LustreError: 304941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913140 with bad export cookie 9938142055457984477 [ 4560.999623] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.006164] LustreError: 304941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4561.157289] Lustre: server umount lustre-MDT0001 complete [ 4564.810725] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:19:04 (1788913144) [ 4565.447887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4566.200347] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:19:05 (1788913145) [ 4567.499387] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4578.982833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.141951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4579.158817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4579.249247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.251567] Lustre: Skipped 1 previous similar message [ 4580.935733] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4584.669432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.772510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4584.788145] Lustre: 310769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.867835] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4604.010605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.128790] Lustre: Denying initial registration attempt from nid 192.168.203.134@tcp, specified as failover [ 4604.133645] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4604.138271] LustreError: 312218:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 4604.142548] LustreError: 312218:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4604.145101] LustreError: 312218:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4604.191707] Lustre: server umount lustre-OST0000 complete [ 4604.194553] LustreError: 312218:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4607.969639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.971793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.978888] Lustre: Skipped 1 previous similar message [ 4607.987240] Lustre: Skipped 1 previous similar message [ 4613.659074] Lustre: server umount lustre-MDT0000 complete [ 4615.132188] LustreError: 311507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913194 with bad export cookie 9938142055457985359 [ 4615.135709] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.137038] LustreError: 311507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4615.266165] Lustre: server umount lustre-MDT0001 complete [ 4619.803128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4621.082150] Key type lgssc unregistered [ 4621.220690] LNet: 313338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4621.225665] LNetError: 313338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4621.234706] LNet: Removed LNI 192.168.203.134@tcp [ 4621.615732] Key type .llcrypt unregistered [ 4621.617117] Key type ._llcrypt unregistered [ 4626.947256] Key type ._llcrypt registered [ 4626.950452] Key type .llcrypt registered [ 4627.004981] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4627.410995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4627.421133] alg: No test for adler32 (adler32-zlib) [ 4628.345872] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4628.497711] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4630.120164] Key type lgssc registered [ 4630.629651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4658.115812] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4661.957978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4661.966496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.089157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4663.101103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4663.169249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.079778] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4669.307538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.378942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4669.398624] Lustre: 316072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4669.536824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4671.390039] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4674.918261] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4677.950177] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.504699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.803954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4697.932864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.045953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4713.793804] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4716.517687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.526036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.539421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4716.553547] Lustre: Skipped 1 previous similar message [ 4721.650239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.667557] Lustre: Skipped 1 previous similar message [ 4722.718065] Lustre: server umount lustre-OST0000 complete [ 4730.849502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4730.862841] LustreError: Skipped 1 previous similar message [ 4730.874038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.921203] Lustre: Skipped 1 previous similar message [ 4730.940315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.887327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.008392] Lustre: server umount lustre-MDT0000 complete [ 4741.570325] LustreError: 316065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913321 with bad export cookie 10503114362177919592 [ 4741.578828] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.595049] LustreError: 316065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4742.546726] Lustre: server umount lustre-MDT0001 complete [ 4756.486641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4759.323348] Key type lgssc unregistered [ 4759.556088] LNet: 319645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4759.565099] LNetError: 319645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4759.588026] LNet: Removed LNI 192.168.203.134@tcp [ 4760.356739] Key type .llcrypt unregistered [ 4760.364090] Key type ._llcrypt unregistered [ 4780.832612] Key type ._llcrypt registered [ 4780.839108] Key type .llcrypt registered [ 4780.975748] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4782.087455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4782.117318] alg: No test for adler32 (adler32-zlib) [ 4783.168800] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4783.366670] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4785.034384] Key type lgssc registered [ 4786.269767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4793.372343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4793.395822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.725111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4798.727122] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4805.089921] LustreError: 321038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4805.100684] LustreError: 321038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4806.078911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.350318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4810.400995] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4816.890176] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4821.707206] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4829.424902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4829.729208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.362673] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4842.250403] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.156172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.161381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.178175] Lustre: Skipped 2 previous similar messages [ 4850.185485] Lustre: Skipped 2 previous similar messages [ 4854.093690] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 4855.267973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4855.274511] Lustre: Skipped 2 previous similar messages [ 4855.454844] Lustre: server umount lustre-MDT0000 complete [ 4858.517272] LustreError: 321018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913438 with bad export cookie 6340520971660820462 [ 4858.523748] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.527577] LustreError: 321018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4859.189130] LustreError: 323245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.251911] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 4860.385223] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.385639] LustreError: 321032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.394201] LustreError: 321034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.395450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4863.427329] Lustre: Skipped 1 previous similar message [ 4864.878910] Lustre: server umount lustre-MDT0001 complete [ 4870.448523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4877.787162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.940841] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x57fe0e3aeec21c1f [ 4883.948939] Lustre: MGC192.168.203.134@tcp: Connection restored to 0@lo (at 0@lo) [ 4884.313703] LustreError: 322868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.328911] LustreError: 322868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4884.377086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.434245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4888.339313] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4893.604108] LustreError: 324631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.607458] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4893.618145] LustreError: 324631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4894.941966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.246675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4899.041331] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4900.327654] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4905.221855] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4906.634857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4911.402905] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4912.710351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4920.289711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4920.305534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.332413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.343446] Lustre: Skipped 1 previous similar message [ 4923.379930] Lustre: server umount lustre-OST0000 complete [ 4931.042149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.043058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4931.082332] Lustre: Skipped 1 previous similar message [ 4941.281599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.283167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4941.286920] Lustre: Skipped 6 previous similar messages [ 4941.522550] Lustre: server umount lustre-MDT0000 complete [ 4944.758534] LustreError: 321017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913524 with bad export cookie 6340520971660827679 [ 4944.762427] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.766967] LustreError: 321017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4945.049986] Lustre: server umount lustre-MDT0001 complete [ 4954.726817] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4957.350805] Key type lgssc unregistered [ 4957.669923] LNet: 327305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4957.688209] LNetError: 327305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4957.707847] LNet: Removed LNI 192.168.203.134@tcp [ 4958.686385] Key type .llcrypt unregistered [ 4958.688982] Key type ._llcrypt unregistered [ 4978.956747] Key type ._llcrypt registered [ 4978.960567] Key type .llcrypt registered [ 4979.056698] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4980.112703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4980.134174] alg: No test for adler32 (adler32-zlib) [ 4981.246192] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 4981.460544] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4983.120216] Key type lgssc registered [ 4983.973959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.969610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.986491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.291493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.906765] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5001.712441] LustreError: 328709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.744971] LustreError: 328709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5002.611102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.005241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5006.542120] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5012.642732] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5016.360848] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5018.594389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5018.600514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.612049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5021.665928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.666437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.576565] Lustre: server umount lustre-MDT0000 complete [ 5026.789794] LustreError: 328709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.806363] LustreError: 328709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5027.094423] LustreError: 329444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913606 with bad export cookie 8320705670842361512 [ 5027.095484] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.101535] LustreError: 329444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5027.426838] Lustre: server umount lustre-MDT0001 complete [ 5037.724965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.013782] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5038.038388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5038.215398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.330590] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5047.534714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.590844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5047.611568] Lustre: 331298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5047.788253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.820330] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5056.181712] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5060.005144] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5067.302060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5067.431357] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5067.438751] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5067.445231] LustreError: 333176:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5067.450738] LustreError: 333176:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5067.454165] LustreError: 333176:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5067.502926] Lustre: server umount lustre-OST0000 complete [ 5067.508744] LustreError: 333176:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5075.500817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.691218] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5075.846433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.977384] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5085.152980] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5085.311656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5090.699075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5090.796034] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5090.850819] Lustre: lustre-OST0001: new disk, initializing [ 5090.853938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5090.858749] Lustre: 334718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5090.919875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5093.289638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5093.298104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5093.388261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5095.064556] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5100.056480] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.595199] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.596428] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.614993] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5103.620490] Lustre: Skipped 2 previous similar messages [ 5108.271915] Lustre: server umount lustre-OST0001 complete [ 5108.712129] LustreError: 333644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.728489] LustreError: 333644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5111.266254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5111.276325] LustreError: Skipped 1 previous similar message [ 5111.279135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.286298] Lustre: Skipped 1 previous similar message [ 5111.289740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.840969] LustreError: 333643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.860225] LustreError: 333643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5118.949953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.950518] LustreError: 333644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.960056] Lustre: Skipped 3 previous similar messages [ 5125.089629] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5125.235628] Lustre: server umount lustre-OST0000 complete [ 5129.186677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.189167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5129.195203] Lustre: Skipped 2 previous similar messages [ 5129.211053] Lustre: Skipped 3 previous similar messages [ 5134.241080] Lustre: server umount lustre-MDT0000 complete [ 5134.307132] LustreError: 332451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.320088] LustreError: 332451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5136.627539] LustreError: 333670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913716 with bad export cookie 8320705670842362058 [ 5136.630177] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.636813] LustreError: 333670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5136.804360] Lustre: server umount lustre-MDT0001 complete [ 5143.953524] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5145.921313] Key type lgssc unregistered [ 5146.123851] LNet: 336676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5146.136985] LNetError: 336676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5146.161550] LNet: Removed LNI 192.168.203.134@tcp [ 5146.756857] Key type .llcrypt unregistered [ 5146.759602] Key type ._llcrypt unregistered [ 5195.916704] Key type ._llcrypt registered [ 5195.919775] Key type .llcrypt registered [ 5196.023950] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5206.484097] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5207.286336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5207.304476] alg: No test for adler32 (adler32-zlib) [ 5208.316665] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 5208.537640] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5210.176167] Key type lgssc registered [ 5211.342947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5235.603939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5235.618709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.833688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5236.880285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5236.973565] Lustre: lustre-MDT0000: new disk, initializing [ 5237.048033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.085779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5239.865467] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5247.108172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.197612] Lustre: 341580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5247.238153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5247.242076] Lustre: Skipped 1 previous similar message [ 5247.294522] Lustre: lustre-MDT0001: new disk, initializing [ 5247.360591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.388809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5247.398989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5250.516946] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5257.298384] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5260.637691] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5265.056740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.235111] Lustre: lustre-OST0000: new disk, initializing [ 5265.239149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5265.245732] Lustre: 343549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5265.316776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.378492] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5275.166690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5275.172981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5275.236229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5276.573151] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5279.863355] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5279.991882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5283.114850] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5283.263793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5285.345477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.348379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.356088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.365356] Lustre: Skipped 1 previous similar message [ 5285.370613] Lustre: Skipped 1 previous similar message [ 5290.468164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.471905] Lustre: Skipped 1 previous similar message [ 5290.604555] Lustre: server umount lustre-OST0000 complete [ 5293.537239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5293.540938] LustreError: Skipped 1 previous similar message [ 5293.543342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.558496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.587817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.588682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.707438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.711590] Lustre: Skipped 2 previous similar messages [ 5307.360122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5307.491294] Lustre: server umount lustre-MDT0000 complete [ 5309.868121] LustreError: 341573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913889 with bad export cookie 14417528895019973242 [ 5309.872271] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.875765] LustreError: 341573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5310.101273] Lustre: server umount lustre-MDT0001 complete [ 5315.896206] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:31:34 (1788913894) [ 5320.460358] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:31:39 (1788913899) [ 5326.269145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.626515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.307731] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5331.939197] LustreError: 346304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.955156] LustreError: 346304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5334.107436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.307211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5336.721614] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5341.063879] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.404858] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5348.219872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.453429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.007225] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5356.146283] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5365.933455] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5366.978576] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5367.892916] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5368.774135] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5369.782523] Lustre: Failing over lustre-MDT0000 [ 5369.975748] Lustre: server umount lustre-MDT0000 complete [ 5370.339498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5370.342525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.350418] LustreError: 346303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.759371] LustreError: 347345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.435247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.442131] Lustre: Skipped 1 previous similar message [ 5378.866460] LustreError: 347345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.878100] LustreError: 347345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5383.984571] LustreError: 346298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.992360] LustreError: 346298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5384.353481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.434792] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.579795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.617791] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5386.809674] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5389.106414] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5389.800989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5389.816751] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5389.839267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5392.403592] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.227150] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5395.510345] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5396.587431] Lustre: Failing over lustre-MDT0000 [ 5396.770947] Lustre: server umount lustre-MDT0000 complete [ 5399.371995] LustreError: 348790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.038783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5400.040856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.057440] Lustre: Skipped 2 previous similar messages [ 5411.284286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.370141] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.536235] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5413.468758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5414.705310] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5416.947670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5416.951991] Lustre: Skipped 2 previous similar messages [ 5416.977189] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5417.009663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5420.083520] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5421.037453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5427.169414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5427.171997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.174952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.184706] Lustre: Skipped 2 previous similar messages [ 5427.194586] Lustre: Skipped 4 previous similar messages [ 5430.615292] Lustre: server umount lustre-MDT0000 complete [ 5432.293136] LustreError: 348936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.302048] LustreError: 348936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5432.690415] LustreError: 346283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914012 with bad export cookie 14417528895019975909 [ 5432.693748] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5432.697798] LustreError: 346283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5432.844316] Lustre: server umount lustre-MDT0001 complete [ 5445.250333] Lustre: server umount lustre-OST0000 complete [ 5452.971301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.290736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.294458] Lustre: Skipped 1 previous similar message [ 5455.324402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5459.575565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.813140] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5465.421941] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5467.965276] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5471.537560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.877770] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5477.866621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5478.416332] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5493.217443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5493.223851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.232483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5496.799588] Lustre: server umount lustre-OST0000 complete [ 5504.713653] Lustre: server umount lustre-MDT0000 complete [ 5505.506229] LustreError: 352633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.513593] LustreError: 352633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5506.405269] LustreError: 352614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914085 with bad export cookie 14417528895019976658 [ 5506.407965] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.414594] LustreError: 352614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5506.558077] Lustre: server umount lustre-MDT0001 complete [ 5511.096569] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5512.392422] Key type lgssc unregistered [ 5512.550776] LNet: 356184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5512.554548] LNetError: 356184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.576965] LNet: Removed LNI 192.168.203.134@tcp [ 5513.994703] Key type .llcrypt unregistered [ 5513.996452] Key type ._llcrypt unregistered [ 5527.684699] Key type ._llcrypt registered [ 5527.685950] Key type .llcrypt registered [ 5527.749644] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5528.476150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5528.537878] alg: No test for adler32 (adler32-zlib) [ 5529.498602] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 5529.863891] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5531.520179] Key type lgssc registered [ 5532.810697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5541.299458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5541.318311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.803494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.705972] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5553.235162] LustreError: 357677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.266124] LustreError: 357677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5558.009601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.308903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5563.553472] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5581.325601] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.061247] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5608.162183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.994982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5613.131524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5617.475303] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5627.054980] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5642.729218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.732421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.758176] Lustre: Skipped 2 previous similar messages [ 5642.761775] Lustre: Skipped 1 previous similar message [ 5647.048694] Lustre: server umount lustre-MDT0000 complete [ 5647.843145] LustreError: 358775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.868184] LustreError: 358775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5651.852241] LustreError: 357657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914231 with bad export cookie 14708498707895092458 [ 5651.871790] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5651.879033] LustreError: 357657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5652.508496] Lustre: server umount lustre-MDT0001 complete [ 5666.921661] Lustre: server umount lustre-OST0000 complete [ 5678.524335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.141442] LustreError: 361415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.200961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.954980] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5684.197857] LustreError: 361416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.316775] LustreError: 361415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.755208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.102434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5696.121451] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5703.463318] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5708.455444] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5715.025621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.411755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5721.601499] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5723.645793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5728.529244] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5743.589747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5743.598342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.616264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.619201] Lustre: Skipped 1 previous similar message [ 5745.935949] Lustre: server umount lustre-MDT0000 complete [ 5749.172540] LustreError: 361396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914328 with bad export cookie 14708498707895093578 [ 5749.184311] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5749.186220] LustreError: 361396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5749.220843] LustreError: 361411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.223484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.254362] Lustre: Skipped 2 previous similar messages [ 5749.265202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5749.277645] Lustre: Skipped 3 previous similar messages [ 5749.536922] Lustre: server umount lustre-MDT0001 complete [ 5755.141682] Lustre: server umount lustre-OST0000 complete [ 5769.597660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.037456] LustreError: 365323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.136406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.801494] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5783.238275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.401379] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5795.436427] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5800.824327] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5807.835684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.184093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.190209] Lustre: Skipped 1 previous similar message [ 5814.929287] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5815.354076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5822.545306] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.606800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.617717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.639558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.629387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5833.359813] Lustre: server umount lustre-OST0000 complete [ 5839.841133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5839.853512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.877759] Lustre: Skipped 1 previous similar message [ 5839.888313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.891301] Lustre: Skipped 2 previous similar messages [ 5843.843106] Lustre: server umount lustre-MDT0000 complete [ 5845.994110] LustreError: 365621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.006713] LustreError: 365621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5847.063279] LustreError: 365305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914426 with bad export cookie 14708498707895094600 [ 5847.070780] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.076871] LustreError: 365305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5847.541890] Lustre: server umount lustre-MDT0001 complete [ 5860.509547] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5863.632375] Key type lgssc unregistered [ 5863.968992] LNet: 368857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5863.984106] LNetError: 368857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5863.999146] LNet: Removed LNI 192.168.203.134@tcp [ 5865.204365] Key type .llcrypt unregistered [ 5865.209662] Key type ._llcrypt unregistered [ 5891.493843] Key type ._llcrypt registered [ 5891.500171] Key type .llcrypt registered [ 5891.612725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5893.145145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5893.168816] alg: No test for adler32 (adler32-zlib) [ 5894.263607] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 5894.580841] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5896.336353] Key type lgssc registered [ 5897.303448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5905.458227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5905.470914] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.477697] LustreError: 370292:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5905.496092] LustreError: 370292:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5905.503578] LustreError: 370292:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5905.516554] LustreError: 370292:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5905.531855] LustreError: 370292:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5913.363828] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.368842] LustreError: 370690:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5913.382483] LustreError: 370690:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5913.388979] LustreError: 370690:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5913.393655] LustreError: 370690:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5913.399237] LustreError: 370690:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5931.818949] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5934.613472] Key type lgssc unregistered [ 5934.931274] LNet: 371692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5934.944728] LNetError: 371692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5934.987257] LNet: Removed LNI 192.168.203.134@tcp [ 5935.759795] Key type .llcrypt unregistered [ 5935.766118] Key type ._llcrypt unregistered [ 5962.169684] Key type ._llcrypt registered [ 5962.174638] Key type .llcrypt registered [ 5962.384831] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5976.710033] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5977.807607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5977.825990] alg: No test for adler32 (adler32-zlib) [ 5978.857338] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 5979.190619] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5980.880237] Key type lgssc registered [ 5982.178235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6015.879887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6015.901269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.160281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6017.215474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6017.377560] Lustre: lustre-MDT0000: new disk, initializing [ 6017.540455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.598749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6021.776770] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6031.081189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.184765] Lustre: 374845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6031.220106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6031.223135] Lustre: Skipped 1 previous similar message [ 6031.300267] Lustre: lustre-MDT0001: new disk, initializing [ 6031.407962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6031.452564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6031.472848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6035.159826] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6043.056388] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6047.915248] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6053.487699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.707418] Lustre: lustre-OST0000: new disk, initializing [ 6053.711105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6053.718034] Lustre: 376815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6053.801404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.010236] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6061.125901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6061.140160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6061.257271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6067.143054] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6071.084790] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6071.280377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6074.887990] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6075.051674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6076.901021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.907349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.919297] Lustre: Skipped 1 previous similar message [ 6076.922609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.934317] Lustre: Skipped 1 previous similar message [ 6082.022517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.026051] Lustre: Skipped 1 previous similar message [ 6082.653386] Lustre: server umount lustre-OST0000 complete [ 6087.142072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.143637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.153460] Lustre: Skipped 1 previous similar message [ 6087.171704] Lustre: Skipped 1 previous similar message [ 6091.686761] Lustre: server umount lustre-MDT0000 complete [ 6094.463701] LustreError: 374838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914674 with bad export cookie 13850792204478919474 [ 6094.472929] LustreError: 374838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6094.479613] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.818372] Lustre: server umount lustre-MDT0001 complete [ 6101.288590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6102.617743] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:44:41 (1788914681) [ 6108.394108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.749224] LustreError: 379272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.761917] LustreError: 379272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6108.834203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.979347] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6114.276326] LustreError: 379273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.416604] LustreError: 379272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.738618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.256564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6126.490379] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6133.870312] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6139.592705] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6146.821453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.280211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.177128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6161.606585] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6169.346107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6169.486348] Lustre: lustre-OST0001: new disk, initializing [ 6169.493624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6169.499247] Lustre: 382110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6169.561567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6175.478435] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6177.422544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6177.443914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6177.524846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6187.979720] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.123150] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6193.126293] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.146446] LustreError: Skipped 1 previous similar message [ 6193.157743] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6193.172605] Lustre: Skipped 1 previous similar message [ 6198.257269] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6198.261542] Lustre: Skipped 3 previous similar messages [ 6198.905474] Lustre: server umount lustre-OST0001 complete [ 6200.161812] LustreError: 381095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.481353] LustreError: 381097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.503937] LustreError: 381097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6217.187154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.191218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.197570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.743145] LustreError: 381098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.779025] LustreError: 381098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6222.137773] Lustre: server umount lustre-OST0000 complete [ 6228.961754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.961872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6228.983447] Lustre: Skipped 1 previous similar message [ 6234.083362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6234.091411] Lustre: Skipped 5 previous similar messages [ 6240.226507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6240.472698] Lustre: server umount lustre-MDT0000 complete [ 6244.207406] LustreError: 379253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914823 with bad export cookie 13850792204478920363 [ 6244.209239] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.234775] LustreError: 379253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6250.743660] Lustre: server umount lustre-MDT0001 complete [ 6260.486439] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6263.419058] Key type lgssc unregistered [ 6263.785891] LNet: 384097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6263.799758] LNetError: 384097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6263.835939] LNet: Removed LNI 192.168.203.134@tcp [ 6264.797737] Key type .llcrypt unregistered [ 6264.802546] Key type ._llcrypt unregistered [ 6316.185901] Key type ._llcrypt registered [ 6316.188841] Key type .llcrypt registered [ 6316.314540] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6317.221032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6317.262787] alg: No test for adler32 (adler32-zlib) [ 6318.292624] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 6318.547993] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 6320.256164] Key type lgssc registered [ 6321.551467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6328.602452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.623897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.877620] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6329.899313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6330.084501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.051783] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6342.402742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.530508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6342.577833] Lustre: 386893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6342.794717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.152869] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6354.525385] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6360.219805] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.106450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6363.119484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.154767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.767044] Lustre: server umount lustre-MDT0000 complete [ 6372.372365] LustreError: 386884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914951 with bad export cookie 17971481735426599916 [ 6372.381522] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.384629] LustreError: 386884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6372.817689] Lustre: server umount lustre-MDT0001 complete [ 6383.655745] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:49:21 (1788914961) [ 6390.655606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6391.442115] LustreError: 389407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.466501] LustreError: 389407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6391.512851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.576569] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6396.901340] LustreError: 389408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.022460] LustreError: 389407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.282823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.532077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6410.297958] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6417.310842] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6422.831465] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6429.170624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.311686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6429.497269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.183061] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6441.926735] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.901925] Lustre: Permanently deactivating lustre-OST0000 [ 6448.933072] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6456.868440] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6458.403850] LustreError: 392390:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6458.412952] LustreError: 392390:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6458.420042] LustreError: 392390:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6461.282179] LustreError: 392440:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6461.296599] LustreError: 392440:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6469.660348] Lustre: server umount lustre-OST0000 complete [ 6472.201303] LustreError: 392689:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6472.211882] LustreError: 392689:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6476.263838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.275144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.289815] Lustre: Skipped 2 previous similar messages [ 6476.296320] Lustre: Skipped 1 previous similar message [ 6481.381470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.389281] Lustre: Skipped 2 previous similar messages [ 6484.534479] Lustre: server umount lustre-MDT0000 complete [ 6486.498453] LustreError: 389402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.520082] LustreError: 389402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6487.872963] LustreError: 389388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915067 with bad export cookie 17971481735426600497 [ 6487.881547] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6487.884481] LustreError: 389388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6498.397937] Lustre: server umount lustre-MDT0001 complete [ 6507.260977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.996583] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6515.000484] LustreError: 393858:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6521.701448] Lustre: 394197:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.134@tcp [ 6523.780148] Lustre: server umount MGS complete [ 6535.465453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.956767] LustreError: 394921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.973437] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6535.982830] Lustre: Skipped 1 previous similar message [ 6536.034370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.799930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6546.401815] LustreError: 394922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.427877] LustreError: 394922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6550.698731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.036852] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6555.757363] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6564.678766] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.636847] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6573.895579] Lustre: Permanently reactivating lustre-OST0000 [ 6573.918428] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6573.923245] Lustre: Skipped 2 previous similar messages [ 6600.162553] LustreError: 396654:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6607.712631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6608.158972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.161966] Lustre: Skipped 1 previous similar message [ 6616.194788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6624.728734] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.884432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.889709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6630.899604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6630.911692] Lustre: Skipped 1 previous similar message [ 6630.934231] LustreError: Skipped 1 previous similar message [ 6632.363906] Lustre: server umount lustre-OST0000 complete [ 6638.049816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6638.056808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.072403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6638.076791] Lustre: Skipped 1 previous similar message [ 6642.473249] Lustre: server umount lustre-MDT0000 complete [ 6646.245638] LustreError: 395650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.267598] LustreError: 395650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6646.371952] LustreError: 394901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915225 with bad export cookie 17971481735426601547 [ 6646.384575] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.390043] LustreError: 394901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6646.702806] Lustre: server umount lustre-MDT0001 complete [ 6655.691485] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6658.419881] Key type lgssc unregistered [ 6658.698749] LNet: 399235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6658.711868] LNetError: 399235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6658.730629] LNet: Removed LNI 192.168.203.134@tcp [ 6659.833871] Key type .llcrypt unregistered [ 6659.837550] Key type ._llcrypt unregistered [ 6679.412518] Key type ._llcrypt registered [ 6679.416274] Key type .llcrypt registered [ 6679.562853] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 6694.898779] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6695.968791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6696.091808] alg: No test for adler32 (adler32-zlib) [ 6697.226720] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 6697.443248] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 6699.048116] Key type lgssc registered [ 6700.044703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6735.825326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6735.857336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.238423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6737.308209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6737.395993] Lustre: lustre-MDT0000: new disk, initializing [ 6737.515791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.545321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6742.358961] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6753.233352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.387929] Lustre: 402387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6753.454208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6753.458817] Lustre: Skipped 1 previous similar message [ 6753.542789] Lustre: lustre-MDT0001: new disk, initializing [ 6753.676085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6753.730054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6753.736418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6759.335968] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6771.178146] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.344095] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6785.021039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.334314] Lustre: lustre-OST0000: new disk, initializing [ 6785.338206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6785.346625] Lustre: 404358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6785.511454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6792.307793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6792.318060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6792.506099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6793.171155] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6803.968847] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6808.964926] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6809.204036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6814.447970] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6814.731990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6818.278728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.279342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.325347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.328928] Lustre: Skipped 1 previous similar message [ 6818.349156] Lustre: Skipped 1 previous similar message [ 6823.102608] Lustre: server umount lustre-OST0000 complete [ 6828.513847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.525096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.555174] Lustre: Skipped 1 previous similar message [ 6833.633908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6833.651647] Lustre: Skipped 2 previous similar messages [ 6834.098816] Lustre: server umount lustre-MDT0000 complete [ 6838.010205] LustreError: 402380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915417 with bad export cookie 10804663686491158287 [ 6838.020417] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.033768] LustreError: 402380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6838.630621] Lustre: server umount lustre-MDT0001 complete [ 6847.210741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6849.189860] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:57:07 (1788915427) [ 6856.376581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.970788] LustreError: 406821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.990813] LustreError: 406821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6857.118581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.327780] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6862.307449] LustreError: 406822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.325641] LustreError: 406822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6867.437673] LustreError: 406821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.705478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.069369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6874.674894] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6882.472903] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6888.617172] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.455036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.749758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.123761] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6911.193607] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.362143] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6917.284063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6923.745328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.754575] LustreError: Skipped 1 previous similar message [ 6923.762972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.786368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6925.802727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6925.815093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.080442] Lustre: server umount lustre-OST0000 complete [ 6936.043723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.088863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6936.116319] Lustre: Skipped 3 previous similar messages [ 6939.732771] Lustre: server umount lustre-MDT0000 complete [ 6941.155469] LustreError: 406821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.178275] LustreError: 406821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6944.944443] LustreError: 406803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915524 with bad export cookie 10804663686491159169 [ 6944.951154] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.956829] LustreError: 406803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6945.557554] Lustre: server umount lustre-MDT0001 complete [ 6959.161728] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6963.007252] Key type lgssc unregistered [ 6963.334653] LNet: 410544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6963.356196] LNetError: 410544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6963.385234] LNet: Removed LNI 192.168.203.134@tcp [ 6964.235475] Key type .llcrypt unregistered [ 6964.237743] Key type ._llcrypt unregistered [ 6990.830782] Key type ._llcrypt registered [ 6990.833717] Key type .llcrypt registered [ 6990.937755] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6991.247341] Key type .llcrypt unregistered [ 6991.252793] Key type ._llcrypt unregistered [ 7009.727580] Key type ._llcrypt registered [ 7009.730126] Key type .llcrypt registered [ 7009.814924] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7010.556926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7010.587845] alg: No test for adler32 (adler32-zlib) [ 7011.629395] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7011.920318] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7013.656934] Key type lgssc registered [ 7014.843429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7022.890402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7022.931107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.526176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.757989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7034.867665] LustreError: 412748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.901310] LustreError: 412748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7039.153873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.542772] LustreError: 412749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.594853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7044.596137] LustreError: 412748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.619737] LustreError: 412748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7046.899164] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7049.700758] LustreError: 412745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.714772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7049.721969] LustreError: 412745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7054.171500] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7054.822547] LustreError: 412749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7054.848811] LustreError: 412749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7060.971430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.180889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7066.182514] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7076.840771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.842717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.867810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.882660] Lustre: Skipped 1 previous similar message [ 7081.956509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7081.973713] Lustre: Skipped 1 previous similar message [ 7083.066993] Lustre: server umount lustre-OST0000 complete [ 7089.989850] Lustre: server umount lustre-MDT0000 complete [ 7092.201473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7092.205814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.220579] LustreError: 412749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.236260] Lustre: Skipped 2 previous similar messages [ 7092.266024] LustreError: 412749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7094.440097] LustreError: 414943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915674 with bad export cookie 4627198991030991129 [ 7094.445995] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.457012] LustreError: 414943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7095.030162] Lustre: server umount lustre-MDT0001 complete [ 7108.651099] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7112.030859] Key type lgssc unregistered [ 7112.431941] LNet: 415908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7112.439322] LNetError: 415908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7112.465656] LNet: Removed LNI 192.168.203.134@tcp [ 7113.423254] Key type .llcrypt unregistered [ 7113.430359] Key type ._llcrypt unregistered [ 7139.236462] Key type ._llcrypt registered [ 7139.239428] Key type .llcrypt registered [ 7139.370866] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7140.467802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7140.504940] alg: No test for adler32 (adler32-zlib) [ 7141.827375] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7142.142195] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7143.890228] Key type lgssc registered [ 7145.358922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7153.968115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7153.993262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.632397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7174.753383] LustreError: 417296:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 7201.504243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.153862] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7216.449475] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7224.375749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.128464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.207164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7230.440879] LustreError: 417323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.483896] LustreError: 417323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7230.642196] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7235.562452] LustreError: 418332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.593134] LustreError: 418332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7240.687577] LustreError: 418334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.722801] LustreError: 418334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7241.208409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.758815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7247.076589] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7255.310558] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7262.649637] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7271.393661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7271.397841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.422971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.424977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.429846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.436178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.461267] Lustre: Skipped 1 previous similar message [ 7276.693842] Lustre: server umount lustre-OST0000 complete [ 7282.659482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.664378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.676627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7287.187842] Lustre: server umount lustre-MDT0000 complete [ 7287.781945] LustreError: 418338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.814875] LustreError: 418338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7291.453653] LustreError: 417304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915871 with bad export cookie 10225313716497092266 [ 7291.458669] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.471367] LustreError: 417304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7291.899209] Lustre: server umount lustre-MDT0001 complete [ 7303.998519] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7307.394480] Key type lgssc unregistered [ 7307.788916] LNet: 420866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7307.796974] LNetError: 420866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7307.821804] LNet: Removed LNI 192.168.203.134@tcp [ 7308.973735] Key type .llcrypt unregistered [ 7308.975577] Key type ._llcrypt unregistered [ 7339.571356] Key type ._llcrypt registered [ 7339.573379] Key type .llcrypt registered [ 7339.792347] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7341.474834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7341.557839] alg: No test for adler32 (adler32-zlib) [ 7342.998888] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7343.424499] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7345.120165] Key type lgssc registered [ 7346.562072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7356.096100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7356.147696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.908257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.367902] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7368.175412] LustreError: 422301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.202863] LustreError: 422301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7373.057136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.284855] LustreError: 422302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.520777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7378.078505] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7385.980329] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7392.555670] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7399.791791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7400.189070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.258117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7406.902111] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7415.350545] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.873653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7419.886353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.910368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7422.301468] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 7422.948568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.061529] Lustre: server umount lustre-MDT0000 complete [ 7427.413338] LustreError: 422298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.277778] LustreError: 422283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.34@tcp arrived at 1788916008 with bad export cookie 7732123824833230947 [ 7429.289967] LustreError: 422283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7431.496980] LustreError: 422284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788916011 with bad export cookie 7732123824833230058 [ 7431.535895] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.543710] LustreError: 422284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7433.186037] LustreError: 423320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.186613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.187150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.187155] Lustre: Skipped 1 previous similar message [ 7433.188267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.188271] Lustre: Skipped 3 previous similar messages [ 7433.195280] LustreError: 423320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7436.195939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7437.870563] Lustre: server umount lustre-OST0000 complete [ 7441.381219] LustreError: 423320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.410837] LustreError: 423320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7445.598503] Lustre: server umount lustre-MDT0001 complete [ 7458.574617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7461.787190] Key type lgssc unregistered [ 7462.068156] LNet: 425983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7462.084357] LNetError: 425983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7462.109824] LNet: Removed LNI 192.168.203.134@tcp [ 7463.092223] Key type .llcrypt unregistered [ 7463.095046] Key type ._llcrypt unregistered [ 7489.367526] Key type ._llcrypt registered [ 7489.368887] Key type .llcrypt registered [ 7489.503437] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7490.342524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7490.361910] alg: No test for adler32 (adler32-zlib) [ 7491.500292] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7491.831627] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7493.601098] Key type lgssc registered [ 7495.213936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7504.052711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7504.071634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.525862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7511.365819] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7516.134241] LustreError: 427397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.157331] LustreError: 427397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7520.232161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.746830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7525.817306] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7534.906758] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.036733] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7548.310822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.713438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7550.765620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7556.424774] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7564.493818] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7571.427102] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.451509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7571.456535] Lustre: Skipped 1 previous similar message [ 7572.332806] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7575.501561] Lustre: server umount lustre-MDT0001 complete [ 7576.547357] LustreError: 428125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.670658] LustreError: 427392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.695883] LustreError: 427392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7586.786829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.787559] LustreError: 427397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7586.797598] Lustre: Skipped 1 previous similar message [ 7586.804145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.445959] Lustre: server umount lustre-OST0000 complete [ 7591.911682] LustreError: 428125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.919945] Lustre: server umount lustre-MDT0000 complete [ 7610.537853] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7613.876616] Key type lgssc unregistered [ 7614.141224] LNet: 431080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7614.157410] LNetError: 431080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7614.182984] LNet: Removed LNI 192.168.203.134@tcp [ 7615.283852] Key type .llcrypt unregistered [ 7615.285604] Key type ._llcrypt unregistered [ 7641.796575] Key type ._llcrypt registered [ 7641.801964] Key type .llcrypt registered [ 7641.944822] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 7656.970651] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7657.866803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.882922] alg: No test for adler32 (adler32-zlib) [ 7658.948528] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7659.368761] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7661.112194] Key type lgssc registered [ 7662.325637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7707.655804] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7724.411274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7724.439165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7725.768978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7725.796536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7725.899873] Lustre: lustre-MDT0000: new disk, initializing [ 7726.045704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.075064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7730.564610] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7746.217444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.539139] Lustre: lustre-OST0000: new disk, initializing [ 7746.544029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7746.546702] Lustre: Skipped 1 previous similar message [ 7746.552855] Lustre: 436116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7746.635335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.604243] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7754.838152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7754.847203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7754.879632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7762.091695] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7777.049362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.197762] Lustre: 435097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7777.310370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7777.388391] Lustre: lustre-MDT0001: new disk, initializing [ 7777.487140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7777.530537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7777.542286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7782.383940] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7791.132025] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7791.303978] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7795.110598] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7795.353920] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7800.295208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.309167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7800.317058] Lustre: Skipped 1 previous similar message [ 7803.240474] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7805.413653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7806.225695] Lustre: server umount lustre-MDT0001 complete [ 7808.347698] LustreError: 435103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7808.375290] LustreError: 435103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7810.535226] LustreError: 435103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.567605] LustreError: 435103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7815.652218] LustreError: 435104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.699439] LustreError: 435104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7820.773585] LustreError: 436573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.791387] LustreError: 436573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7825.891956] LustreError: 435107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.896741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.935554] Lustre: Skipped 1 previous similar message [ 7825.945551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7825.961771] Lustre: Skipped 1 previous similar message [ 7829.700769] Lustre: server umount lustre-OST0000 complete [ 7839.623986] Lustre: server umount lustre-MDT0000 complete [ 7851.525692] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7853.879518] Key type lgssc unregistered [ 7854.181622] LNet: 439859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7854.194469] LNetError: 439859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7854.216563] LNet: Removed LNI 192.168.203.134@tcp [ 7854.938063] Key type .llcrypt unregistered [ 7854.939476] Key type ._llcrypt unregistered [ 7904.114475] Key type ._llcrypt registered [ 7904.116519] Key type .llcrypt registered [ 7904.204596] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7904.568373] Key type .llcrypt unregistered [ 7904.570538] Key type ._llcrypt unregistered [ 7921.782526] Key type ._llcrypt registered [ 7921.785533] Key type .llcrypt registered [ 7921.861606] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7923.111491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7923.265937] alg: No test for adler32 (adler32-zlib) [ 7924.351279] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 7924.576224] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7926.289160] Key type lgssc registered [ 7927.507379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7956.297613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7956.325218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.582319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7957.607672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7957.705148] Lustre: lustre-MDT0000: new disk, initializing [ 7957.851711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.886533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7962.339192] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7972.447749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.571564] Lustre: 443227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7972.639377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7972.646875] Lustre: Skipped 1 previous similar message [ 7972.726451] Lustre: lustre-MDT0001: new disk, initializing [ 7972.866446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7972.910596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7972.924396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7977.759637] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7987.868484] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7994.185984] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8001.646374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.293658] Lustre: lustre-OST0000: new disk, initializing [ 8002.309222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8002.324287] Lustre: 445197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8002.527379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.773341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8008.788993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8008.980021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8009.194481] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8019.229151] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.154680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8029.175416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.225239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.409500] Lustre: server umount lustre-MDT0000 complete [ 8039.916869] LustreError: 443240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.952044] LustreError: 443240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8040.504635] LustreError: 443219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788916620 with bad export cookie 6058190351248048147 [ 8040.506900] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8040.512950] LustreError: 443219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8040.871618] Lustre: server umount lustre-MDT0001 complete [ 8055.567163] Lustre: server umount lustre-OST0000 complete [ 8066.356716] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:17:25 (1788916645) [ 8073.260721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8073.708563] LustreError: 447227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.799830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8077.523402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8078.821035] LustreError: 447228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.939155] LustreError: 447227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8084.864931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.395699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8091.515243] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8101.430426] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8108.280902] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8130.499064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8130.937458] Lustre: Found index 0 for lustre-OST0000, updating log [ 8131.231125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.046779] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8144.516962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8148.614341] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8160.231430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8160.254797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.292526] Lustre: Skipped 2 previous similar messages [ 8160.302990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8160.309136] Lustre: Skipped 2 previous similar messages [ 8161.382872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8161.388894] Lustre: Skipped 2 previous similar messages [ 8162.717358] Lustre: server umount lustre-OST0000 complete [ 8182.755910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8182.765688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.777708] Lustre: Skipped 1 previous similar message [ 8182.784610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.223871] Lustre: server umount lustre-MDT0000 complete [ 8188.387654] LustreError: 447223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.414330] LustreError: 447223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8189.173707] LustreError: 447207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788916768 with bad export cookie 6058190351248049547 [ 8189.178262] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.181943] LustreError: 447207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8189.516660] Lustre: server umount lustre-MDT0001 complete [ 8197.427281] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:19:35 (1788916775) [ 8200.224151] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [ 8807.074625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.322686] LustreError: 453636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.367055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.791535] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8812.513735] LustreError: 453637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.863046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.109670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8817.557429] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8821.640928] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8824.937184] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8841.830448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.960201] Lustre: Found index 0 for lustre-OST0000, updating log [ 8842.122304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8845.351290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8849.273986] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8850.511201] Lustre: server umount lustre-OST0000 complete [ 8856.035421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8856.042443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.049139] Lustre: Skipped 1 previous similar message [ 8856.052258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8856.054927] Lustre: Skipped 2 previous similar messages [ 8857.570682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.573793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.582779] Lustre: Skipped 1 previous similar message [ 8858.765858] Lustre: server umount lustre-MDT0000 complete [ 8860.624454] LustreError: 453617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788917440 with bad export cookie 6058190351248050625 [ 8860.628768] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8860.632037] LustreError: 453617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8860.799295] Lustre: server umount lustre-MDT0001 complete [ 8865.824238] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [ 8880.533355] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8886.006590] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8904.428581] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:31:23 (1788917483) [ 8912.760442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8914.364935] Key type lgssc unregistered [ 8914.545897] LNet: 460249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8914.551397] LNetError: 460249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8971.944663] LNet: Removed LNI 192.168.203.134@tcp [ 8972.447512] Key type .llcrypt unregistered [ 8972.449780] Key type ._llcrypt unregistered [ 8981.138749] Key type ._llcrypt registered [ 8981.140258] Key type .llcrypt registered [ 8981.198392] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8981.706252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.727411] alg: No test for adler32 (adler32-zlib) [ 8982.675248] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 8982.831323] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8984.448165] Key type lgssc registered [ 8985.025687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8987.176087] LNet: Added LNI 192.168.203.134@tcp100 [8/256/0/180] [ 8987.662346] LNet: Added LNI 192.168.203.134@tcp101 [8/256/0/180] [ 8988.103649] LNet: Added LNI 192.168.203.134@tcp102 [8/256/0/180] [ 8988.553414] LNet: Added LNI 192.168.203.134@tcp103 [8/256/0/180] [ 8989.012570] LNet: Added LNI 192.168.203.134@tcp104 [8/256/0/180] [ 8989.443464] LNet: Added LNI 192.168.203.134@tcp105 [8/256/0/180] [ 8989.852233] LNet: Added LNI 192.168.203.134@tcp106 [8/256/0/180] [ 8990.265459] LNet: Added LNI 192.168.203.134@tcp107 [8/256/0/180] [ 8990.683809] LNet: Added LNI 192.168.203.134@tcp108 [8/256/0/180] [ 8991.065855] LNet: Added LNI 192.168.203.134@tcp109 [8/256/0/180] [ 8991.470714] LNet: Added LNI 192.168.203.134@tcp110 [8/256/0/180] [ 8991.892112] LNet: Added LNI 192.168.203.134@tcp111 [8/256/0/180] [ 8992.319795] LNet: Added LNI 192.168.203.134@tcp112 [8/256/0/180] [ 8992.790290] LNet: Added LNI 192.168.203.134@tcp113 [8/256/0/180] [ 8993.237397] LNet: Added LNI 192.168.203.134@tcp114 [8/256/0/180] [ 8993.655416] LNet: Added LNI 192.168.203.134@tcp115 [8/256/0/180] [ 8994.107709] LNet: Added LNI 192.168.203.134@tcp116 [8/256/0/180] [ 8994.562358] LNet: Added LNI 192.168.203.134@tcp117 [8/256/0/180] [ 8995.001636] LNet: Added LNI 192.168.203.134@tcp118 [8/256/0/180] [ 8995.384943] LNet: Added LNI 192.168.203.134@tcp119 [8/256/0/180] [ 8995.826263] LNet: Added LNI 192.168.203.134@tcp120 [8/256/0/180] [ 8996.259190] LNet: Added LNI 192.168.203.134@tcp121 [8/256/0/180] [ 8996.680834] LNet: Added LNI 192.168.203.134@tcp122 [8/256/0/180] [ 8997.181106] LNet: Added LNI 192.168.203.134@tcp123 [8/256/0/180] [ 8997.668049] LNet: Added LNI 192.168.203.134@tcp124 [8/256/0/180] [ 8998.158107] LNet: Added LNI 192.168.203.134@tcp125 [8/256/0/180] [ 8998.638797] LNet: Added LNI 192.168.203.134@tcp126 [8/256/0/180] [ 8999.107245] LNet: Added LNI 192.168.203.134@tcp127 [8/256/0/180] [ 8999.544374] LNet: Added LNI 192.168.203.134@tcp128 [8/256/0/180] [ 8999.974313] LNet: Added LNI 192.168.203.134@tcp129 [8/256/0/180] [ 9000.522468] LNet: Added LNI 192.168.203.134@tcp130 [8/256/0/180] [ 9000.986918] LNet: Added LNI 192.168.203.134@tcp131 [8/256/0/180] [ 9001.447688] LNet: Added LNI 192.168.203.134@tcp132 [8/256/0/180] [ 9001.895442] LNet: Added LNI 192.168.203.134@tcp133 [8/256/0/180] [ 9002.329412] LNet: Added LNI 192.168.203.134@tcp134 [8/256/0/180] [ 9002.781882] LNet: Added LNI 192.168.203.134@tcp135 [8/256/0/180] [ 9005.841887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9005.850526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.903974] Lustre: 463241:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [ 9006.941029] LNetError: 460704:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9006.950192] LNetError: Protocol error connecting to 192.168.203.134@tcp120 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9006.956968] LNetError: Skipped 6 previous similar messages [ 9007.026166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9007.042322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9007.083728] Lustre: lustre-MDT0000: new disk, initializing [ 9007.124522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9007.135199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9007.394945] LNet: 7 local NIs in recovery (showing 7): 192.168.203.134@tcp125, 192.168.203.134@tcp126, 192.168.203.134@tcp123, 192.168.203.134@tcp127, 192.168.203.134@tcp128, 192.168.203.134@tcp129, 192.168.203.134@tcp120 [ 9008.962991] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9009.443591] LNetError: 463270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9009.450742] LNetError: 463270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [ 9009.455511] LNetError: Protocol error connecting to 192.168.203.134@tcp126 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9009.461346] LNetError: Skipped 18 previous similar messages [ 9013.537115] LNetError: 460706:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9013.542358] LNetError: 460706:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 9013.545825] LNetError: Protocol error connecting to 192.168.203.134@tcp126 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9013.840848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9013.909405] Lustre: 463258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9013.942507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9013.945426] Lustre: Skipped 1 previous similar message [ 9013.986727] Lustre: lustre-MDT0001: new disk, initializing [ 9014.001485] Lustre: 464061:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [ 9014.005875] Lustre: 464061:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9014.060759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9014.082366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9014.087313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9015.851529] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9020.771345] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9021.729311] LNetError: 460705:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9021.735601] LNetError: Protocol error connecting to 192.168.203.134@tcp126 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9023.556517] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9026.874487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9027.017958] Lustre: lustre-OST0000: new disk, initializing [ 9027.021128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9027.024504] Lustre: 465232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9027.066053] Lustre: 465237:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [ 9027.070925] Lustre: 465237:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 9027.101956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9029.899765] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9035.181181] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9036.280548] Lustre: 465990:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [ 9036.287929] Lustre: 465990:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9036.326132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9036.333386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9036.403300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9037.092436] LNetError: 463261:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9037.103328] LNetError: Protocol error connecting to 192.168.203.134@tcp126 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9041.377835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.380860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9041.385149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9041.390579] Lustre: Skipped 1 previous similar message [ 9045.229773] Lustre: server umount lustre-OST0000 complete [ 9046.306722] LNetError: 463270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9046.313137] LNetError: 463270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9046.317280] LNetError: Protocol error connecting to 192.168.203.134@tcp109 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9046.322748] LNetError: Skipped 6 previous similar messages [ 9050.081110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9050.089611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.096925] Lustre: Skipped 1 previous similar message [ 9050.100769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.619070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.623328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.630068] Lustre: Skipped 1 previous similar message [ 9056.737934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.740898] Lustre: Skipped 1 previous similar message [ 9061.344795] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9061.444632] Lustre: server umount lustre-MDT0000 complete [ 9063.339269] LustreError: 463250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788917642 with bad export cookie 14709606690185677189 [ 9063.342632] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9063.351780] LustreError: 463250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9063.522702] Lustre: server umount lustre-MDT0001 complete [ 9063.714315] LNetError: 463261:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [ 9063.721169] LNetError: 463261:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 9063.726380] LNetError: Protocol error connecting to 192.168.203.134@tcp123 at host 192.168.203.134:988: is it running a compatible version of Lustre? [ 9063.736481] LNetError: Skipped 10 previous similar messages [ 9066.784358] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp126 [ 9068.953682] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9070.438837] Key type lgssc unregistered [ 9070.689886] LNet: 467144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.695721] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9070.701218] LNet: Removed LNI 192.168.203.134@tcp [ 9071.712179] LNet: Removed LNI 192.168.203.134@tcp100 [ 9071.715458] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9071.720631] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9071.725260] LNet: Removed LNI 192.168.203.134@tcp101 [ 9072.736360] LNet: Removed LNI 192.168.203.134@tcp102 [ 9072.739273] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9072.744151] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9072.748186] LNet: Removed LNI 192.168.203.134@tcp103 [ 9073.760126] LNet: Removed LNI 192.168.203.134@tcp104 [ 9074.784236] LNet: Removed LNI 192.168.203.134@tcp105 [ 9074.787036] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9074.794685] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9074.802126] LNet: Removed LNI 192.168.203.134@tcp106 [ 9075.808429] LNet: Removed LNI 192.168.203.134@tcp107 [ 9076.832150] LNet: Removed LNI 192.168.203.134@tcp108 [ 9077.856147] LNet: Removed LNI 192.168.203.134@tcp109 [ 9078.880173] LNet: Removed LNI 192.168.203.134@tcp110 [ 9078.885744] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9078.889842] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9078.893385] LNet: Removed LNI 192.168.203.134@tcp111 [ 9079.904150] LNet: Removed LNI 192.168.203.134@tcp112 [ 9080.928167] LNet: Removed LNI 192.168.203.134@tcp113 [ 9081.952169] LNet: Removed LNI 192.168.203.134@tcp114 [ 9082.976336] LNet: Removed LNI 192.168.203.134@tcp115 [ 9084.000344] LNet: Removed LNI 192.168.203.134@tcp116 [ 9085.024146] LNet: Removed LNI 192.168.203.134@tcp117 [ 9086.048208] LNet: Removed LNI 192.168.203.134@tcp118 [ 9087.072168] LNet: Removed LNI 192.168.203.134@tcp119 [ 9087.073684] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9087.079004] LNetError: 467144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9087.084575] LNet: Removed LNI 192.168.203.134@tcp120 [ 9088.096139] LNet: Removed LNI 192.168.203.134@tcp121 [ 9089.120496] LNet: Removed LNI 192.168.203.134@tcp122 [ 9090.145717] LNet: Removed LNI 192.168.203.134@tcp123 [ 9091.168162] LNet: Removed LNI 192.168.203.134@tcp124 [ 9092.192170] LNet: Removed LNI 192.168.203.134@tcp125 [ 9092.195482] LNet: Removed LNI 192.168.203.134@tcp126 [ 9093.216307] LNet: Removed LNI 192.168.203.134@tcp127 [ 9094.240264] LNet: Removed LNI 192.168.203.134@tcp128 [ 9095.264194] LNet: Removed LNI 192.168.203.134@tcp129 [ 9095.266689] LNet: Removed LNI 192.168.203.134@tcp130 [ 9095.270956] LNet: Removed LNI 192.168.203.134@tcp131 [ 9095.273558] LNet: Removed LNI 192.168.203.134@tcp132 [ 9095.276262] LNet: Removed LNI 192.168.203.134@tcp133 [ 9095.278566] LNet: Removed LNI 192.168.203.134@tcp134 [ 9095.295603] LNet: Removed LNI 192.168.203.134@tcp135 [ 9095.673300] Key type .llcrypt unregistered [ 9095.674494] Key type ._llcrypt unregistered [ 9106.821435] Key type ._llcrypt registered [ 9106.823368] Key type .llcrypt registered [ 9106.881599] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9114.570562] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9115.048034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9115.071541] alg: No test for adler32 (adler32-zlib) [ 9115.982175] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 9116.104975] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9117.728164] Key type lgssc registered [ 9118.287861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9133.665892] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:35:12 (1788917712) [ 9141.602246] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9142.995985] Key type lgssc unregistered [ 9143.151170] LNet: 471201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9143.155989] LNetError: 471201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9143.178575] LNet: Removed LNI 192.168.203.134@tcp [ 9143.596542] Key type .llcrypt unregistered [ 9143.598532] Key type ._llcrypt unregistered [ 9152.185605] Key type ._llcrypt registered [ 9152.187637] Key type .llcrypt registered [ 9152.247740] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9152.672789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9152.728430] alg: No test for adler32 (adler32-zlib) [ 9153.630824] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 9153.746627] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9155.360183] Key type lgssc registered [ 9155.929648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9160.784980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9160.803790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.946842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9161.963172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9162.012383] Lustre: lustre-MDT0000: new disk, initializing [ 9162.063258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.074857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9164.177752] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9169.615511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.672681] Lustre: 472388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9169.706081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9169.709191] Lustre: Skipped 1 previous similar message [ 9169.752530] Lustre: lustre-MDT0001: new disk, initializing [ 9169.810473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9169.830074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9169.838774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9171.847440] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9176.989858] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9179.830371] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9183.151847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.303826] Lustre: lustre-OST0000: new disk, initializing [ 9183.307600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9183.313958] Lustre: 474355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9183.368978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.731977] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9192.387137] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9192.981145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9192.985691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9193.044642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9194.594795] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lnet_if_list [ 9197.046993] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9197.061106] LNet: Added LNI 192.168.203.134@tcp10 [8/256/0/180] [ 9199.690372] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9199.713350] LNet: Added LNI 192.168.203.134@tcp11 [8/256/0/180] [ 9201.888371] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9201.904735] LNet: Added LNI 192.168.203.134@tcp12 [8/256/0/180] [ 9203.948238] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9203.968339] LNet: Added LNI 192.168.203.134@tcp13 [8/256/0/180] [ 9205.954765] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9205.969036] LNet: Added LNI 192.168.203.134@tcp14 [8/256/0/180] [ 9208.189570] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9208.206462] LNet: Added LNI 192.168.203.134@tcp15 [8/256/0/180] [ 9210.356824] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9210.374519] LNet: Added LNI 192.168.203.134@tcp16 [8/256/0/180] [ 9212.596276] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9212.612456] LNet: Added LNI 192.168.203.134@tcp17 [8/256/0/180] [ 9214.961989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9214.981439] LNet: Added LNI 192.168.203.134@tcp18 [8/256/0/180] [ 9217.380840] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9217.406713] LNet: Added LNI 192.168.203.134@tcp19 [8/256/0/180] [ 9219.807969] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9219.828537] LNet: Added LNI 192.168.203.134@tcp20 [8/256/0/180] [ 9222.230564] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9222.245458] LNet: Added LNI 192.168.203.134@tcp21 [8/256/0/180] [ 9224.620720] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9224.637142] LNet: Added LNI 192.168.203.134@tcp22 [8/256/0/180] [ 9227.268694] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9227.285552] LNet: Added LNI 192.168.203.134@tcp23 [8/256/0/180] [ 9229.781707] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9229.798752] LNet: Added LNI 192.168.203.134@tcp24 [8/256/0/180] [ 9232.372196] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9232.390911] LNet: Added LNI 192.168.203.134@tcp25 [8/256/0/180] [ 9235.168068] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9235.183160] LNet: Added LNI 192.168.203.134@tcp26 [8/256/0/180] [ 9237.989990] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9238.005623] LNet: Added LNI 192.168.203.134@tcp27 [8/256/0/180] [ 9240.745323] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9240.768547] LNet: Added LNI 192.168.203.134@tcp28 [8/256/0/180] [ 9243.328286] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9243.343568] LNet: Added LNI 192.168.203.134@tcp29 [8/256/0/180] [ 9246.006249] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9246.022828] LNet: Added LNI 192.168.203.134@tcp30 [8/256/0/180] [ 9248.878693] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9248.894797] LNet: Added LNI 192.168.203.134@tcp31 [8/256/0/180] [ 9251.667098] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9251.680600] LNet: Added LNI 192.168.203.134@tcp32 [8/256/0/180] [ 9254.867867] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9254.891564] LNet: Added LNI 192.168.203.134@tcp33 [8/256/0/180] [ 9257.933637] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9257.950032] LNet: Added LNI 192.168.203.134@tcp34 [8/256/0/180] [ 9261.060585] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9261.076757] LNet: Added LNI 192.168.203.134@tcp35 [8/256/0/180] [ 9263.955573] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9263.967811] LNet: Added LNI 192.168.203.134@tcp36 [8/256/0/180] [ 9266.915717] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9266.930451] LNet: Added LNI 192.168.203.134@tcp37 [8/256/0/180] [ 9270.001546] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9270.017916] LNet: Added LNI 192.168.203.134@tcp38 [8/256/0/180] [ 9273.221888] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9273.241535] LNet: Added LNI 192.168.203.134@tcp39 [8/256/0/180] [ 9276.289676] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9276.304651] LNet: Added LNI 192.168.203.134@tcp40 [8/256/0/180] [ 9279.400316] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9279.415413] LNet: Added LNI 192.168.203.134@tcp41 [8/256/0/180] [ 9282.509294] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9282.524750] LNet: Added LNI 192.168.203.134@tcp42 [8/256/0/180] [ 9285.682385] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9285.698497] LNet: Added LNI 192.168.203.134@tcp43 [8/256/0/180] [ 9288.914564] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9288.934597] LNet: Added LNI 192.168.203.134@tcp44 [8/256/0/180] [ 9292.082092] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9292.098044] LNet: Added LNI 192.168.203.134@tcp45 [8/256/0/180] [ 9295.330459] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9295.345576] LNet: Added LNI 192.168.203.134@tcp46 [8/256/0/180] [ 9298.655859] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9298.673363] LNet: Added LNI 192.168.203.134@tcp47 [8/256/0/180] [ 9302.004597] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9302.020741] LNet: Added LNI 192.168.203.134@tcp48 [8/256/0/180] [ 9305.423247] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9305.438616] LNet: Added LNI 192.168.203.134@tcp49 [8/256/0/180] [ 9309.026247] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9309.045460] LNet: Added LNI 192.168.203.134@tcp50 [8/256/0/180] [ 9318.368938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9318.372476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.379813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9320.930193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9320.930722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.933237] Lustre: Skipped 1 previous similar message [ 9320.937895] Lustre: Skipped 1 previous similar message [ 9323.683195] Lustre: server umount lustre-MDT0000 complete [ 9325.117165] LustreError: 472424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788917904 with bad export cookie 766624599061197441 [ 9325.121101] LustreError: 472424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9325.122223] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9325.248337] Lustre: server umount lustre-MDT0001 complete [ 9333.330390] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9336.768532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.176162] Lustre: 471661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788917905/real 1788917905] req@00000000277306e6 x1875816200897024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788917921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9342.188417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.440145] Lustre: 471658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788917909/real 1788917909] req@00000000d95058a2 x1875816200897152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788917925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9346.464140] Lustre: 471658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788917910/real 1788917910] req@00000000b261cdc8 x1875816200897280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788917926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9350.368423] LustreError: 471657:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cc1fa8ce x1875816200898176/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9350.502647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9350.608075] Lustre: 471660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788917914/real 1788917914] req@000000006230a815 x1875816200897536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788917930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9352.082318] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9355.404700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.562271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.576161] Lustre: 471660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788917919/real 1788917919] req@000000005f312d1e x1875816200897664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788917935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9356.586585] Lustre: 471660:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9357.160872] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9359.996568] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9360.681404] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9362.135316] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9371.107354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9371.109564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9371.111480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9371.111485] Lustre: Skipped 1 previous similar message [ 9371.112630] LustreError: Skipped 1 previous similar message [ 9371.116695] Lustre: Skipped 1 previous similar message [ 9376.227865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9376.230595] Lustre: Skipped 2 previous similar messages [ 9376.421457] Lustre: server umount lustre-OST0000 complete [ 9381.345758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9381.346417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.347381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.356400] Lustre: Skipped 1 previous similar message [ 9384.098488] Lustre: server umount lustre-MDT0000 complete [ 9385.676186] LustreError: 472424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788917965 with bad export cookie 766624599061199632 [ 9385.680351] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.681721] LustreError: 472424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9385.816583] Lustre: server umount lustre-MDT0001 complete [ 9390.018385] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9391.256609] Key type lgssc unregistered [ 9391.443139] LNet: 501317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9391.447095] LNetError: 501317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9391.450523] LNet: Removed LNI 192.168.203.134@tcp [ 9391.452653] LNet: Removed LNI 192.168.203.134@tcp10 [ 9391.454562] LNet: Removed LNI 192.168.203.134@tcp11 [ 9391.456662] LNet: Removed LNI 192.168.203.134@tcp12 [ 9391.458846] LNet: Removed LNI 192.168.203.134@tcp13 [ 9391.463619] LNet: Removed LNI 192.168.203.134@tcp14 [ 9391.465375] LNet: Removed LNI 192.168.203.134@tcp15 [ 9391.467417] LNet: Removed LNI 192.168.203.134@tcp16 [ 9391.469243] LNet: Removed LNI 192.168.203.134@tcp17 [ 9391.471881] LNet: Removed LNI 192.168.203.134@tcp18 [ 9391.474583] LNet: Removed LNI 192.168.203.134@tcp19 [ 9391.477163] LNet: Removed LNI 192.168.203.134@tcp20 [ 9391.479065] LNet: Removed LNI 192.168.203.134@tcp21 [ 9391.480855] LNet: Removed LNI 192.168.203.134@tcp22 [ 9391.482967] LNet: Removed LNI 192.168.203.134@tcp23 [ 9391.485110] LNet: Removed LNI 192.168.203.134@tcp24 [ 9391.486698] LNet: Removed LNI 192.168.203.134@tcp25 [ 9391.488429] LNet: Removed LNI 192.168.203.134@tcp26 [ 9391.490467] LNet: Removed LNI 192.168.203.134@tcp27 [ 9391.492498] LNet: Removed LNI 192.168.203.134@tcp28 [ 9391.494560] LNet: Removed LNI 192.168.203.134@tcp29 [ 9391.496542] LNet: Removed LNI 192.168.203.134@tcp30 [ 9391.498465] LNet: Removed LNI 192.168.203.134@tcp31 [ 9391.500193] LNet: Removed LNI 192.168.203.134@tcp32 [ 9391.502410] LNet: Removed LNI 192.168.203.134@tcp33 [ 9391.504319] LNet: Removed LNI 192.168.203.134@tcp34 [ 9391.506339] LNet: Removed LNI 192.168.203.134@tcp35 [ 9391.508404] LNet: Removed LNI 192.168.203.134@tcp36 [ 9391.510556] LNet: Removed LNI 192.168.203.134@tcp37 [ 9391.514167] LNet: Removed LNI 192.168.203.134@tcp38 [ 9391.516476] LNet: Removed LNI 192.168.203.134@tcp39 [ 9391.518309] LNet: Removed LNI 192.168.203.134@tcp40 [ 9391.520218] LNet: Removed LNI 192.168.203.134@tcp41 [ 9391.522302] LNet: Removed LNI 192.168.203.134@tcp42 [ 9391.524346] LNet: Removed LNI 192.168.203.134@tcp43 [ 9391.526216] LNet: Removed LNI 192.168.203.134@tcp44 [ 9391.528185] LNet: Removed LNI 192.168.203.134@tcp45 [ 9391.530171] LNet: Removed LNI 192.168.203.134@tcp46 [ 9391.532148] LNet: Removed LNI 192.168.203.134@tcp47 [ 9391.533881] LNet: Removed LNI 192.168.203.134@tcp48 [ 9391.535801] LNet: Removed LNI 192.168.203.134@tcp49 [ 9391.543847] LNet: Removed LNI 192.168.203.134@tcp50 [ 9391.871360] Key type .llcrypt unregistered [ 9391.872881] Key type ._llcrypt unregistered [ 9399.163550] Key type ._llcrypt registered [ 9399.165060] Key type .llcrypt registered [ 9399.209366] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9405.920487] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9406.306054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9406.353847] alg: No test for adler32 (adler32-zlib) [ 9407.234592] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 9407.335486] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9408.936132] Key type lgssc registered [ 9409.347247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9423.356137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9423.362657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9424.465182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9424.478433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9424.519377] Lustre: lustre-MDT0000: new disk, initializing [ 9424.546885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.554565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9426.154931] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9431.028666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.080081] Lustre: 504481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9431.109315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9431.112552] Lustre: Skipped 1 previous similar message [ 9431.151676] Lustre: lustre-MDT0001: new disk, initializing [ 9431.194534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9431.213852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9431.217633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9432.901233] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9437.415908] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9439.709735] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9442.801616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9442.886875] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9442.920131] Lustre: lustre-OST0000: new disk, initializing [ 9442.922541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9442.926533] Lustre: 506501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9442.963752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9444.419300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9444.424705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9444.468739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9445.295545] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9449.550366] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9454.562726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9454.567566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9454.567835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9454.573585] Lustre: Skipped 1 previous similar message [ 9458.588477] Lustre: server umount lustre-OST0000 complete [ 9462.240925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9462.244264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.249247] Lustre: Skipped 1 previous similar message [ 9462.251413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.801417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.801953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.209612] Lustre: server umount lustre-MDT0000 complete [ 9467.647392] LustreError: 504474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788918047 with bad export cookie 13764278954472408460 [ 9467.650823] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9467.652130] LustreError: 504474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9467.776079] Lustre: server umount lustre-MDT0001 complete [ 9471.860341] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9473.004423] Key type lgssc unregistered [ 9473.138480] LNet: 508403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9473.142491] LNetError: 508403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9473.150307] LNet: Removed LNI 192.168.203.134@tcp [ 9473.506366] Key type .llcrypt unregistered [ 9473.508048] Key type ._llcrypt unregistered [ 9481.955403] Key type ._llcrypt registered [ 9481.957107] Key type .llcrypt registered [ 9482.003348] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9482.547820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9482.565152] alg: No test for adler32 (adler32-zlib) [ 9483.440704] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 9483.562184] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9485.160140] Key type lgssc registered [ 9485.641562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9488.756829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9488.766609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.940589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.458658] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9494.721937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.857510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9496.358703] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9498.992358] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9501.073725] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9503.529329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9503.660801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.738651] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9508.263399] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9522.978728] loop0: detected capacity change from 0 to 204800000 [ 9523.065931] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9523.072409] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9523.080222] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.401341] loop0: detected capacity change from 0 to 204800000 [ 9525.477976] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.484521] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.493135] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9527.495642] loop0: detected capacity change from 0 to 204800000 [ 9529.633179] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.683578] Lustre: fs15246-OST0000: new disk, initializing [ 9529.685595] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9529.689257] Lustre: 513256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9529.711702] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.866866] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9536.083349] loop1: detected capacity change from 0 to 204800000 [ 9538.097677] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.138890] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9538.166211] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9538.211434] Lustre: fs15246-MDT0000: new disk, initializing [ 9538.262415] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.273348] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9538.315142] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9539.361874] Lustre: 514378:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000071f50a6f x1875816546860672/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788918134 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9539.715889] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9547.533619] LustreError: 514378:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9547.539821] LustreError: 514378:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9547.544912] LustreError: 514378:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9548.264302] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.379538] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.691229] Lustre: server umount fs15246-MDT0000 complete [ 9557.007903] Lustre: server umount fs15246-OST0000 complete [ 9561.057809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9561.057830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.060512] LustreError: Skipped 1 previous similar message [ 9561.067808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9561.071072] Lustre: Skipped 1 previous similar message [ 9563.618366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.621013] Lustre: Skipped 1 previous similar message [ 9566.675921] Lustre: server umount lustre-OST0000 complete [ 9568.738058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.743983] Lustre: Skipped 1 previous similar message [ 9568.747101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.750461] Lustre: Skipped 1 previous similar message [ 9574.037763] Lustre: server umount lustre-MDT0000 complete [ 9575.208121] LustreError: 511486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788918154 with bad export cookie 15835361920580504541 [ 9575.210770] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9575.212890] LustreError: 511486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9575.319816] Lustre: server umount lustre-MDT0001 complete [ 9578.903210] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9579.954464] Key type lgssc unregistered [ 9580.091367] LNet: 517470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9580.094751] LNetError: 517470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9580.102279] LNet: Removed LNI 192.168.203.134@tcp [ 9580.407347] Key type .llcrypt unregistered [ 9580.408935] Key type ._llcrypt unregistered [ 9587.844982] Key type ._llcrypt registered [ 9587.846262] Key type .llcrypt registered [ 9587.887235] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9588.320916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9588.351595] alg: No test for adler32 (adler32-zlib) [ 9589.212387] Lustre: Lustre: Build Version: 2.17.57_86_g8205644 [ 9589.309476] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9590.888192] Key type lgssc registered [ 9591.259569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9604.646393] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9607.998172] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9619.921259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9619.927927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.006878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9621.016047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9621.049699] Lustre: lustre-MDT0000: new disk, initializing [ 9621.071174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.077634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9622.367200] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9626.602703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.671207] Lustre: 521901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9626.703388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9626.705544] Lustre: Skipped 1 previous similar message [ 9626.745940] Lustre: lustre-MDT0001: new disk, initializing [ 9626.782793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9626.795083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9626.798136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9628.251266] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9631.914053] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9633.648040] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9635.882952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.993105] Lustre: lustre-OST0000: new disk, initializing [ 9635.995250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9635.998089] Lustre: 523867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9636.027104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.024878] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9641.682430] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9643.107058] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9644.023835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9644.028624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9644.060027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9644.196156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9645.433421] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9645.503321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9649.121823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9649.123554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.125249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9649.128506] Lustre: Skipped 1 previous similar message [ 9652.249537] Lustre: server umount lustre-OST0000 complete [ 9654.241648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.242379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9654.245652] Lustre: Skipped 1 previous similar message [ 9654.248522] Lustre: Skipped 2 previous similar messages [ 9659.361492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.363269] Lustre: Skipped 1 previous similar message [ 9659.668942] Lustre: server umount lustre-MDT0000 complete [ 9660.849408] LustreError: 522695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788918240 with bad export cookie 8968653698896340997 [ 9660.852248] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9660.852934] LustreError: 522695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9660.965882] Lustre: server umount lustre-MDT0001 complete [ 9668.637722] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9671.898690] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9682.312387] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9490 sec ========== 21:44:21 (1788918261) [ 9682.847772] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:44:22 (1788918262) === [ 9683.924652] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:44:23 (1788918263) === [ 9696.844184] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9697.816355] Key type lgssc unregistered [ 9697.942403] LNet: 530506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9697.944958] LNetError: 530506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9697.956263] LNet: Removed LNI 192.168.203.134@tcp [ 9698.261886] Key type .llcrypt unregistered [ 9698.262976] Key type ._llcrypt unregistered