[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461598232 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002292] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006914] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009148] LSM: Security Framework initializing [ 0.010071] Yama: becoming mindful. [ 0.011055] SELinux: Initializing. [ 0.012080] *** VALIDATE selinux *** [ 0.020783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025697] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] *** VALIDATE tmpfs *** [ 0.030062] *** VALIDATE proc *** [ 0.031251] *** VALIDATE cgroup *** [ 0.032014] *** VALIDATE cgroup2 *** [ 0.033270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035090] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041257] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.043212] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045030] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048018] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051017] ... event mask: 000000070000000f [ 0.052318] rcu: Hierarchical SRCU implementation. [ 0.054328] smp: Bringing up secondary CPUs ... [ 0.055566] x86: Booting SMP configuration: [ 0.056029] .... node #0, CPUs: #1 #2 #3 [ 0.059723] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.218067] node 0 deferred pages initialised in 155ms [ 0.222010] devtmpfs: initialized [ 0.223241] x86/mm: Memory block size: 128MB [ 0.225722] gcov: version magic: 0x41383552 [ 0.226565] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.234305] pinctrl core: initialized pinctrl subsystem [ 0.236126] [ 0.236623] ************************************************************* [ 0.239015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241013] ** ** [ 0.244015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.246013] ** ** [ 0.249012] ** This means that this kernel is built to expose internal ** [ 0.251010] ** IOMMU data structures, which may compromise security on ** [ 0.253014] ** your system. ** [ 0.255010] ** ** [ 0.257013] ** If you see this message and you are not debugging the ** [ 0.260013] ** kernel, report this immediately to your vendor! ** [ 0.262012] ** ** [ 0.264016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267016] ************************************************************* [ 0.270707] NET: Registered protocol family 16 [ 0.272466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.276069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.280013] cpuidle: using governor menu [ 0.281691] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.284552] PCI: Using configuration type 1 for base access [ 0.286108] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.299112] cryptd: max_cpu_qlen set to 1000 [ 0.301213] ACPI: Added _OSI(Module Device) [ 0.303014] ACPI: Added _OSI(Processor Device) [ 0.304016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.306017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.310000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.318323] ACPI: Interpreter enabled [ 0.320062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322016] ACPI: Using IOAPIC for interrupt routing [ 0.324097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327336] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.337555] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.343022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.347075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352439] acpiphp: Slot [2] registered [ 0.354185] acpiphp: Slot [5] registered [ 0.355154] acpiphp: Slot [6] registered [ 0.357167] acpiphp: Slot [7] registered [ 0.358094] acpiphp: Slot [8] registered [ 0.360143] acpiphp: Slot [9] registered [ 0.361165] acpiphp: Slot [10] registered [ 0.363209] acpiphp: Slot [3] registered [ 0.365091] acpiphp: Slot [4] registered [ 0.366088] acpiphp: Slot [11] registered [ 0.367111] acpiphp: Slot [12] registered [ 0.369106] acpiphp: Slot [13] registered [ 0.370145] acpiphp: Slot [14] registered [ 0.371081] acpiphp: Slot [15] registered [ 0.372120] acpiphp: Slot [16] registered [ 0.373091] acpiphp: Slot [17] registered [ 0.375130] acpiphp: Slot [18] registered [ 0.376086] acpiphp: Slot [19] registered [ 0.377111] acpiphp: Slot [20] registered [ 0.379108] acpiphp: Slot [21] registered [ 0.380079] acpiphp: Slot [22] registered [ 0.382115] acpiphp: Slot [23] registered [ 0.383126] acpiphp: Slot [24] registered [ 0.385102] acpiphp: Slot [25] registered [ 0.386000] acpiphp: Slot [26] registered [ 0.386000] acpiphp: Slot [27] registered [ 0.388136] acpiphp: Slot [28] registered [ 0.389092] acpiphp: Slot [29] registered [ 0.390078] acpiphp: Slot [30] registered [ 0.392112] acpiphp: Slot [31] registered [ 0.393092] PCI host bridge to bus 0000:00 [ 0.394022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.395015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.397018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.400025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.402023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.405045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.407258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410913] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.413103] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.426023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.431044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.434025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.437028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.440018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.442181] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.444775] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449723] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.455016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.468015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.473026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.479000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.487015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.495011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.513016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.521694] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.528016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.535014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561737] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.567021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.572019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.588020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597262] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.603014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.609015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.628023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.638357] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.653017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.672018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682047] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.692019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.724023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.735059] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.737364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.740323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.742423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.745167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.750050] iommu: Default domain type: Passthrough [ 0.751539] SCSI subsystem initialized [ 0.753206] ACPI: bus type USB registered [ 0.755115] usbcore: registered new interface driver usbfs [ 0.757089] usbcore: registered new interface driver hub [ 0.758099] usbcore: registered new device driver usb [ 0.759163] pps_core: LinuxPPS API ver. 1 registered [ 0.760011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763074] PTP clock support registered [ 0.765058] EDAC MC: Ver: 3.0.0 [ 0.767192] PCI: Using ACPI for IRQ routing [ 0.770139] NetLabel: Initializing [ 0.772018] NetLabel: domain hash size = 128 [ 0.774019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.777086] NetLabel: unlabeled traffic allowed by default [ 0.781165] vgaarb: loaded [ 0.784366] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.792455] clocksource: Switched to clocksource kvm-clock [ 0.890915] VFS: Disk quotas dquot_6.6.0 [ 0.892588] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.896142] *** VALIDATE ramfs *** [ 0.897425] *** VALIDATE hugetlbfs *** [ 0.899173] pnp: PnP ACPI init [ 0.900766] pnp: PnP ACPI: found 6 devices [ 0.916725] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.920343] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.921806] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925042] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.926919] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.928990] NET: Registered protocol family 2 [ 0.930952] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.935402] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.938814] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.946500] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950762] TCP: Hash tables configured (established 65536 bind 65536) [ 0.954460] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.957790] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960893] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964097] NET: Registered protocol family 1 [ 0.966520] RPC: Registered named UNIX socket transport module. [ 0.969284] RPC: Registered udp transport module. [ 0.971549] RPC: Registered tcp transport module. [ 0.973075] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.975037] NET: Registered protocol family 44 [ 0.976657] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.978768] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.980723] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.983105] PCI: CLS 0 bytes, default 64 [ 0.984659] Unpacking initramfs... [ 2.356777] debug: unmapping init [mem 0xffff9bc6bcc54000-0xffff9bc6bffbffff] [ 2.363220] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.365841] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.369608] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.855793] Initialise system trusted keyrings [ 2.857881] Key type blacklist registered [ 2.859922] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.870083] zbud: loaded [ 2.873654] *** VALIDATE nfs *** [ 2.874786] *** VALIDATE nfs4 *** [ 2.876474] pstore: using deflate compression [ 2.879155] Platform Keyring initialized [ 2.983765] NET: Registered protocol family 38 [ 2.985724] Key type asymmetric registered [ 2.987093] Asymmetric key parser 'x509' registered [ 2.988990] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.992048] io scheduler mq-deadline registered [ 2.994870] io scheduler kyber registered [ 2.996277] io scheduler bfq registered [ 2.998247] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.001350] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.004436] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.007354] ACPI: Power Button [PWRF] [ 3.014397] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.022274] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.036298] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.042634] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.056787] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.084482] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.113218] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.117781] Non-volatile memory driver v1.3 [ 3.118771] Linux agpgart interface v0.103 [ 3.147450] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.149650] vda: detected capacity change from 0 to 74686464 [ 3.165022] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.167331] vdb: detected capacity change from 0 to 1073741824 [ 3.181654] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.184430] vdc: detected capacity change from 0 to 2621440000 [ 3.197800] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.200644] vdd: detected capacity change from 0 to 2621440000 [ 3.214550] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.216669] vde: detected capacity change from 0 to 4294967296 [ 3.229207] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231775] vdf: detected capacity change from 0 to 4294967296 [ 3.237835] libphy: Fixed MDIO Bus: probed [ 3.242309] usbcore: registered new interface driver usbserial_generic [ 3.244450] usbserial: USB Serial support registered for generic [ 3.246342] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.250208] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.251904] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.254285] mousedev: PS/2 mouse device common for all mice [ 3.258694] rtc_cmos 00:05: RTC can wake from S4 [ 3.261626] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.264993] rtc_cmos 00:05: registered as rtc0 [ 3.266837] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.267650] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.272298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.273491] intel_pstate: CPU model not supported [ 3.281238] hid: raw HID events driver (C) Jiri Kosina [ 3.283219] usbcore: registered new interface driver usbhid [ 3.285193] usbhid: USB HID core driver [ 3.286505] drop_monitor: Initializing network drop monitor service [ 3.288519] Initializing XFRM netlink socket [ 3.290664] NET: Registered protocol family 10 [ 3.293727] Segment Routing with IPv6 [ 3.295267] NET: Registered protocol family 17 [ 3.297338] mpls_gso: MPLS GSO support [ 3.303434] RAS: Correctable Errors collector initialized. [ 3.305870] AVX version of gcm_enc/dec engaged. [ 3.307834] AES CTR mode by8 optimization enabled [ 3.380229] sched_clock: Marking stable (3380199416, 0)->(4282440330, -902240914) [ 3.383874] registered taskstats version 1 [ 3.386096] Loading compiled-in X.509 certificates [ 3.387649] zswap: loaded using pool lzo/zbud [ 3.410663] Key type big_key registered [ 3.423871] Key type encrypted registered [ 3.425525] ima: No TPM chip found, activating TPM-bypass! [ 3.427869] ima: Allocated hash algorithm: sha1 [ 3.429784] ima: No architecture policies found [ 3.431240] evm: Initialising EVM extended attributes: [ 3.433125] evm: security.selinux [ 3.434431] evm: security.ima [ 3.435529] evm: security.capability [ 3.437097] evm: HMAC attrs: 0x1 [ 3.440172] rtc_cmos 00:05: setting system clock to 2026-08-03 20:07:26 UTC (1785787646) [ 3.445756] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.449034] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.458172] Write protecting the kernel read-only data: 28672k [ 3.462116] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.465158] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.498519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.506110] systemd[1]: Detected virtualization kvm. [ 3.508297] systemd[1]: Detected architecture x86-64. [ 3.510084] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.532612] systemd[1]: No hostname configured. [ 3.534297] systemd[1]: Set hostname to . [ 3.536183] random: systemd: uninitialized urandom read (16 bytes read) [ 3.538094] systemd[1]: Initializing machine ID from random generator. [ 3.573477] random: ln: uninitialized urandom read (6 bytes read) [ 3.656094] random: systemd: uninitialized urandom read (16 bytes read) [ 3.658950] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.663299] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.667069] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.274596] device-mapper: uevent: version 1.0.3 [ 4.276783] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.956322] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.999944] random: fast init done [ 5.024620] scsi host0: ata_piix [ 5.032888] scsi host1: ata_piix [ 5.035212] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.037180] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.387298] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.943057] random: crng init done [ 9.944416] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.369458] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.728868] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.099048] SELinux: Disabled at runtime. [ 12.173866] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.182169] systemd[1]: Detected virtualization kvm. [ 12.184229] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.033960] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.037514] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.043298] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.047620] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.051691] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.059088] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.063758] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 13.145134] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.611977] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.029396] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.147819] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.277084] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.296896] EDAC sbridge: Ver: 1.1.2 [ 16.281437] Key type dns_resolver registered [ 16.683786] NFS: Registering the id_resolver key type [ 16.688770] Key type id_resolver registered [ 16.690143] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg621-server login: [ 49.445815] libcfs: loading out-of-tree module taints kernel. [ 49.476464] Key type ._llcrypt registered [ 49.478852] Key type .llcrypt registered [ 49.545060] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 60.308457] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 61.253353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.262987] alg: No test for adler32 (adler32-zlib) [ 62.351484] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 62.820139] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 64.456089] Key type lgssc registered [ 65.367315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.564300] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.396362] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 97.424858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 97.437276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 98.554518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 98.567876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 98.608637] Lustre: lustre-MDT0000: new disk, initializing [ 98.646514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 98.655024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 100.456747] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.162966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 107.207292] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 107.222021] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 107.223784] Lustre: Skipped 1 previous similar message [ 107.265917] Lustre: lustre-MDT0001: new disk, initializing [ 107.296116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.310750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 107.316069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 108.952485] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.599042] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 116.216655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.328918] Lustre: lustre-OST0000: new disk, initializing [ 116.331791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 116.335573] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 116.366837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.452516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 118.459592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 118.490444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 119.141669] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.835899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 125.893212] Lustre: lustre-OST0001: new disk, initializing [ 125.895721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 125.898951] Lustre: 9490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 125.923199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 128.258858] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.087206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 132.092546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 132.127701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 134.859729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.074961] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 143.865842] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.811253] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing yml_node [ 148.021293] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 149.274317] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 150.412962] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 151.275578] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 3 16:09:54 EDT 2026 [ 159.955742] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 160.713905] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 174.048683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 174.056331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 174.065575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 178.144259] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 178.144601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 178.155532] Lustre: Skipped 3 previous similar messages [ 179.582273] Lustre: server umount lustre-MDT0000 complete [ 181.631970] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787824 with bad export cookie 3639611566194944756 [ 181.633771] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 181.639799] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 183.264755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 183.266399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 183.276214] Lustre: Skipped 3 previous similar messages [ 186.276704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 186.280442] Lustre: Skipped 1 previous similar message [ 187.855287] Lustre: server umount lustre-MDT0001 complete [ 189.758853] Lustre: server umount lustre-OST0000 complete [ 191.620853] Lustre: server umount lustre-OST0001 complete [ 194.803502] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 199.663166] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 217.915365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.060248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.076408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.123644] Lustre: lustre-MDT0000: new disk, initializing [ 218.166872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.179967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.203062] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 225.945316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.011184] Lustre: 15028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.151822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.155309] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 233.505436] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 236.460827] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 239.665243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.832973] Lustre: lustre-OST0000: new disk, initializing [ 239.835088] Lustre: Skipped 1 previous similar message [ 239.837771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.840951] Lustre: Skipped 2 previous similar messages [ 239.844671] Lustre: 16995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.887884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.894592] Lustre: Skipped 1 previous similar message [ 241.428843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.437957] Lustre: Skipped 1 previous similar message [ 241.443527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.490508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.758666] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 247.935555] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 250.082736] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 250.189973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 252.107570] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 252.224663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 256.995605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 256.997871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 257.002652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 257.002661] Lustre: Skipped 1 previous similar message [ 257.006726] Lustre: Skipped 1 previous similar message [ 257.012312] LustreError: Skipped 1 previous similar message [ 259.307106] Lustre: server umount lustre-OST0000 complete [ 262.113054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.114413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.233790] LustreError: 15039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.244944] LustreError: 15039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 267.277675] Lustre: server umount lustre-MDT0000 complete [ 269.131716] LustreError: 15019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787912 with bad export cookie 3639611566194946289 [ 269.135180] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.138808] LustreError: 15019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 269.293898] Lustre: server umount lustre-MDT0001 complete [ 271.209726] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:11:54 (1785787914) [ 272.237920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 272.999503] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:11:56 (1785787916) [ 276.104032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.323615] LustreError: 19401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.354332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.306539] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 281.570241] LustreError: 19402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.478742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.614955] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 288.303675] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 291.108397] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 294.693407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 297.879313] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 301.423494] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 316.384602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 316.387968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.392990] Lustre: Skipped 1 previous similar message [ 316.395743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 316.401995] Lustre: Skipped 3 previous similar messages [ 320.060637] Lustre: server umount lustre-OST0000 complete [ 328.405513] Lustre: server umount lustre-MDT0000 complete [ 330.790132] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787973 with bad export cookie 3639611566194947171 [ 330.791440] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.802111] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 331.004336] Lustre: server umount lustre-MDT0001 complete [ 336.555696] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 337.944681] Key type lgssc unregistered [ 338.106648] LNet: 22956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.109862] LNetError: 22956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.116689] LNet: Removed LNI 192.168.206.121@tcp [ 338.568276] Key type .llcrypt unregistered [ 338.569965] Key type ._llcrypt unregistered [ 349.512599] Key type ._llcrypt registered [ 349.513625] Key type .llcrypt registered [ 349.560811] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 349.999146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 350.011345] alg: No test for adler32 (adler32-zlib) [ 350.914559] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 351.052440] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 352.655753] Key type lgssc registered [ 353.218205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 357.177075] loop: module loaded [ 357.204919] loop0: detected capacity change from 0 to 204800000 [ 357.322440] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 357.333033] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 357.510844] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.148388] loop0: detected capacity change from 0 to 204800000 [ 360.228527] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.238147] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.244984] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 362.870509] loop0: detected capacity change from 0 to 204800000 [ 365.706406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 365.715629] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.834109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 366.868351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 366.919569] Lustre: lustre-MDT0000: new disk, initializing [ 366.963991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.975502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.770829] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 372.810609] loop1: detected capacity change from 0 to 204800000 [ 375.152777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 375.299059] Lustre: lustre-OST07c6: new disk, initializing [ 375.302143] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 375.304939] Lustre: Skipped 1 previous similar message [ 375.308235] Lustre: 26119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 375.346216] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 377.981926] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 381.962353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 381.967485] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 381.987065] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 385.630848] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 385.634876] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 385.638243] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 385.641265] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 385.644878] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 385.647970] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 400.252338] Lustre: 25106:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 400.258194] Lustre: 25106:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 400.264052] Lustre: 25106:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 400.267978] Lustre: 25106:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 400.279914] Lustre: 25106:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 400.284316] Lustre: 25106:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 404.474801] Lustre: 25700:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 404.481938] Lustre: 25700:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 404.486720] Lustre: 25700:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 404.493029] Lustre: 25700:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 404.497256] Lustre: 25700:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 404.507202] Lustre: 25700:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 404.511904] Lustre: 25700:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 404.516654] Lustre: 25700:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 404.521147] Lustre: 25700:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 404.525894] Lustre: 25700:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 404.530924] Lustre: 25700:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 404.535856] Lustre: 25700:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.672787] Lustre: 25106:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 407.684797] Lustre: 25106:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 407.687936] Lustre: 25106:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 407.691837] Lustre: 25106:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.697124] Lustre: 25106:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 407.701487] Lustre: 25106:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.706116] Lustre: 25106:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 407.711987] Lustre: 25106:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.716163] Lustre: 25106:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 407.721691] Lustre: 25106:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.726060] Lustre: 25106:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 407.731045] Lustre: 25106:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 411.806528] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 411.812511] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 411.818993] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 411.825672] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 411.833090] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 411.839528] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 411.846275] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 411.851108] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 411.856549] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 411.862196] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 411.867685] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 411.874699] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.369265] Lustre: 25107:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 422.376837] Lustre: 25107:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 422.382630] Lustre: 25107:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 422.386139] Lustre: 25107:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.389020] Lustre: 25107:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 422.392008] Lustre: 25107:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.395048] Lustre: 25107:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 422.398859] Lustre: 25107:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.403613] Lustre: 25107:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 422.406877] Lustre: 25107:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.416922] Lustre: 25107:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 422.424356] Lustre: 25107:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 468.584331] Lustre: server umount lustre-OST07c6 complete [ 483.384962] Lustre: server umount lustre-MDT0000 complete [ 490.456943] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 491.670166] Key type lgssc unregistered [ 491.836187] LNet: 28673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.840288] LNetError: 28673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.851635] LNet: Removed LNI 192.168.206.121@tcp [ 492.216352] Key type .llcrypt unregistered [ 492.217984] Key type ._llcrypt unregistered [ 500.579955] Key type ._llcrypt registered [ 500.581710] Key type .llcrypt registered [ 500.628921] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 500.970591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 500.985769] alg: No test for adler32 (adler32-zlib) [ 501.862663] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 501.973744] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 503.583164] Key type lgssc registered [ 504.104211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 507.802611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 507.813239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.031218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.835369] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 514.693443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.861119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 516.810507] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 520.118758] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 522.614654] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.699289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.861989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.707562] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 532.417318] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 534.059685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 547.810693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 547.813888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.820412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.345761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.347119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.352186] LustreError: Skipped 1 previous similar message [ 549.354664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.890688] Lustre: server umount lustre-OST0000 complete [ 554.465877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.471410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.473801] Lustre: Skipped 2 previous similar messages [ 559.583930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.587737] Lustre: Skipped 1 previous similar message [ 559.712130] Lustre: server umount lustre-MDT0000 complete [ 561.198267] LustreError: 29900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788204 with bad export cookie 12280476240154693188 [ 561.203307] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.204164] LustreError: 29900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 561.337309] Lustre: server umount lustre-MDT0001 complete [ 565.719985] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 566.948938] Key type lgssc unregistered [ 567.089643] LNet: 33651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.095610] LNetError: 33651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.109912] LNet: Removed LNI 192.168.206.121@tcp [ 567.490630] Key type .llcrypt unregistered [ 567.492490] Key type ._llcrypt unregistered [ 576.261375] Key type ._llcrypt registered [ 576.263242] Key type .llcrypt registered [ 576.312824] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 576.825014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.832389] alg: No test for adler32 (adler32-zlib) [ 577.744785] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 577.889833] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 579.495218] Key type lgssc registered [ 580.007030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 583.658750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 583.670059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.890966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.927433] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 591.363624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.546992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 593.398108] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 597.162170] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.736352] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.753226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.914706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.413139] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 608.438165] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 609.066883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 624.608478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 624.613145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.620186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 626.658894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 626.659702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 626.669133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.063315] Lustre: server umount lustre-OST0000 complete [ 631.776213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.783599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.786500] Lustre: Skipped 2 previous similar messages [ 636.896195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 636.899126] Lustre: Skipped 1 previous similar message [ 642.017149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.020497] Lustre: Skipped 1 previous similar message [ 646.111353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 646.199109] Lustre: server umount lustre-MDT0000 complete [ 647.964365] LustreError: 34878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788291 with bad export cookie 1065245742136776713 [ 647.966952] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.969112] LustreError: 34878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 648.110533] Lustre: server umount lustre-MDT0001 complete [ 652.584112] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 653.876785] Key type lgssc unregistered [ 654.025701] LNet: 38449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.029831] LNetError: 38449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.037461] LNet: Removed LNI 192.168.206.121@tcp [ 654.416155] Key type .llcrypt unregistered [ 654.417821] Key type ._llcrypt unregistered [ 665.161844] Key type ._llcrypt registered [ 665.163631] Key type .llcrypt registered [ 665.210234] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 665.557650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.567818] alg: No test for adler32 (adler32-zlib) [ 666.451723] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 666.548170] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 668.135945] Key type lgssc registered [ 668.586473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 671.863621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 671.872613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.066209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.691090] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 678.328978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.490427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 680.117874] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 683.111870] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.575264] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 688.315792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.476105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.945491] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 694.008812] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 699.752780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 703.968270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.971728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.978335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.992389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.993534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.995226] Lustre: Skipped 1 previous similar message [ 705.002113] Lustre: Skipped 1 previous similar message [ 708.961931] Lustre: server umount lustre-MDT0000 complete [ 710.115338] LustreError: 41812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.123350] LustreError: 41812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 710.384338] LustreError: 39825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788353 with bad export cookie 11925061206341918880 [ 710.387323] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.389334] LustreError: 39825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 710.637628] LustreError: 39842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.647557] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 710.651632] Lustre: Skipped 2 previous similar messages [ 715.232306] LustreError: 39841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.232553] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.246508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 716.585908] Lustre: server umount lustre-MDT0001 complete [ 734.802709] Lustre: server umount lustre-OST0000 complete [ 741.311668] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 742.611980] Key type lgssc unregistered [ 742.753716] LNet: 43701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.757777] LNetError: 43701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.768377] LNet: Removed LNI 192.168.206.121@tcp [ 743.185433] Key type .llcrypt unregistered [ 743.186957] Key type ._llcrypt unregistered [ 754.385526] Key type ._llcrypt registered [ 754.387276] Key type .llcrypt registered [ 754.432591] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 754.807877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.832703] alg: No test for adler32 (adler32-zlib) [ 755.740625] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 755.856146] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 757.447214] Key type lgssc registered [ 757.940811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.158295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 761.177796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.394665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.158459] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 767.729384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.893472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 769.503582] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 772.554456] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 774.671867] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 777.124665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.283597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.496736] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 782.514335] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 787.555356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 798.180486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.181365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 798.186356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 798.188558] LustreError: Skipped 1 previous similar message [ 798.190480] Lustre: Skipped 1 previous similar message [ 799.220434] Lustre: lustre-OST0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 801.180657] Lustre: server umount lustre-OST0000 complete [ 814.278306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.284262] Lustre: Skipped 1 previous similar message [ 814.287312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.401761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.401955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.412178] Lustre: Skipped 1 previous similar message [ 820.520409] Lustre: server umount lustre-MDT0000 complete [ 821.921891] LustreError: 46936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788464 with bad export cookie 9614027345793332561 [ 821.923159] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 821.927394] LustreError: 46936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 828.087950] Lustre: server umount lustre-MDT0001 complete [ 832.095756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 833.312062] Key type lgssc unregistered [ 833.447527] LNet: 48821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.451108] LNetError: 48821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.459414] LNet: Removed LNI 192.168.206.121@tcp [ 833.820839] Key type .llcrypt unregistered [ 833.822477] Key type ._llcrypt unregistered [ 843.673945] Key type ._llcrypt registered [ 843.675777] Key type .llcrypt registered [ 843.732207] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 844.192634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.212977] alg: No test for adler32 (adler32-zlib) [ 845.089559] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 845.209080] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 846.807242] Key type lgssc registered [ 847.244637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.523695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.532382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.778849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.449268] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 856.984075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.188530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.872333] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 861.826652] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.097709] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 866.800427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 866.927164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.233666] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 872.367508] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 876.134219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 880.050805] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 881.321878] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 892.121142] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-0316h22m15s [ 892.738653] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 896.993046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.996404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 896.998327] Lustre: Skipped 2 previous similar messages [ 897.002071] Lustre: Skipped 1 previous similar message [ 898.083892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.088593] Lustre: Skipped 3 previous similar messages [ 899.565659] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 899.569992] Lustre: Skipped 1 previous similar message [ 899.764412] Lustre: server umount lustre-MDT0000 complete [ 901.392566] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 904.164760] LustreError: 51458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.177259] LustreError: 51458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 904.755985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.831351] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.977067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.010816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 907.162625] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 908.820203] Lustre: DEBUG MARKER: Wait for df (36556) ... [ 909.568576] Lustre: DEBUG MARKER: done [ 912.101354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.102750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 912.107919] Lustre: Skipped 1 previous similar message [ 912.113525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.118562] Lustre: Skipped 1 previous similar message [ 917.209378] Lustre: server umount lustre-OST0000 complete [ 922.593028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.593825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.595511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.598737] Lustre: Skipped 1 previous similar message [ 924.828738] Lustre: server umount lustre-MDT0000 complete [ 926.223705] LustreError: 50200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788569 with bad export cookie 4117988095515843671 [ 926.225759] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 926.229955] LustreError: 50200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 926.355691] Lustre: server umount lustre-MDT0001 complete [ 930.369417] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 931.527133] Key type lgssc unregistered [ 931.669398] LNet: 55133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.672490] LNetError: 55133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.679316] LNet: Removed LNI 192.168.206.121@tcp [ 931.995395] Key type .llcrypt unregistered [ 931.997127] Key type ._llcrypt unregistered [ 946.921544] Key type ._llcrypt registered [ 946.923310] Key type .llcrypt registered [ 946.965817] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 947.456900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 947.468085] alg: No test for adler32 (adler32-zlib) [ 948.344245] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 948.443728] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 950.031214] Key type lgssc registered [ 950.437333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 953.368865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 953.376781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.490163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 954.498987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 954.566537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.043965] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 959.342458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.397199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 959.413065] Lustre: 57314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 959.532929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 961.143554] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 964.102507] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 966.130621] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 968.818186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.899615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 968.992943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.252891] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 974.302994] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 976.408827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 984.343515] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-0316h23m47s [ 984.932223] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 987.426284] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 988.292103] Lustre: DEBUG MARKER: Creating a test file and stat it [ 988.308293] LustreError: 57351:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1008.645491] Lustre: lustre-MDT0000: Client fb12c481-fea0-4e8f-a26d-cf2ccf63f210 (at 192.168.206.21@tcp) reconnecting [ 1028.351163] LustreError: 57351:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1029.155625] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1030.053760] Lustre: DEBUG MARKER: done [ 1032.675025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.675488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.678915] LustreError: Skipped 1 previous similar message [ 1032.684872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.688099] Lustre: Skipped 1 previous similar message [ 1037.716038] Lustre: server umount lustre-OST0000 complete [ 1041.376043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1041.378271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.381922] Lustre: Skipped 1 previous similar message [ 1041.383848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.401427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.403961] Lustre: Skipped 2 previous similar messages [ 1045.468426] Lustre: server umount lustre-MDT0000 complete [ 1046.826509] LustreError: 57309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788689 with bad export cookie 1346304173956888168 [ 1046.831428] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.832149] LustreError: 57309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1046.955290] Lustre: server umount lustre-MDT0001 complete [ 1050.684179] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1051.790441] Key type lgssc unregistered [ 1051.950336] LNet: 61640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.954287] LNetError: 61640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.967374] LNet: Removed LNI 192.168.206.121@tcp [ 1052.308686] Key type .llcrypt unregistered [ 1052.310188] Key type ._llcrypt unregistered [ 1070.055030] Key type ._llcrypt registered [ 1070.056197] Key type .llcrypt registered [ 1070.100520] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1070.497087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.542543] alg: No test for adler32 (adler32-zlib) [ 1071.403653] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1071.514142] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1073.095236] Key type lgssc registered [ 1073.530684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.941310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1074.951529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.079274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1076.087225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1076.148767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.534083] Lustre: Failing over lustre-MDT0000 [ 1076.641250] Lustre: server umount lustre-MDT0000 complete [ 1079.078186] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:25:22 (1785788722) [ 1081.561384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.697986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.141951] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1086.224375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.267352] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1086.279935] Lustre: 64434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1086.392037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.975882] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1090.970190] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1093.086392] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1095.687665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1095.779385] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1095.892808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.281778] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1101.070359] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.905943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1109.628359] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1112.031989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1112.035307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.042524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.081293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.083490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.087519] Lustre: Skipped 1 previous similar message [ 1114.094869] Lustre: Skipped 1 previous similar message [ 1118.129472] Lustre: server umount lustre-MDT0000 complete [ 1119.695448] LustreError: 64426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788762 with bad export cookie 335474618381758275 [ 1119.698735] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.702773] LustreError: 64426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1119.848490] Lustre: server umount lustre-MDT0001 complete [ 1121.265454] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1123.024451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1126.246225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.440484] LustreError: 63357:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000021da7a2e x1872535186604672/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1129.554929] LustreError: 66280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.562542] LustreError: 66280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1129.587652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.622368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1131.147699] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1134.385227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.506065] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.089551] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1138.602022] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1139.684227] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1140.568678] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1141.215096] Lustre: 63361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788767/real 1785788767] req@00000000066b5fc6 x1872535186604416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785788783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1144.544574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.546228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.546866] Lustre: Skipped 1 previous similar message [ 1144.553136] Lustre: Skipped 1 previous similar message [ 1145.567143] Lustre: 63361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788772/real 1785788772] req@0000000071c67e0a x1872535186604544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785788788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1148.646753] Lustre: server umount lustre-MDT0000 complete [ 1149.922691] LustreError: 68197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.931637] LustreError: 68197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1150.099680] LustreError: 64424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788793 with bad export cookie 335474618381760543 [ 1150.101143] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.106855] LustreError: 64424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1150.241841] Lustre: server umount lustre-MDT0001 complete [ 1152.751892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1156.109514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.159783] LustreError: 63357:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008efb2100 x1872535186623232/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.275744] LustreError: 66280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.305236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.307861] Lustre: Skipped 1 previous similar message [ 1160.334355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1161.852229] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1164.871276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.983442] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.648271] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1169.366410] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.403845] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1171.375670] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1171.407127] Lustre: 63361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788798/real 1785788798] req@000000002b34e7d9 x1872535186622976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785788814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1175.520797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.523229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.526735] Lustre: Skipped 1 previous similar message [ 1175.529693] Lustre: Skipped 1 previous similar message [ 1175.535488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1176.543143] Lustre: 63358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785788803/real 1785788803] req@00000000245515cb x1872535186623104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785788819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1179.622354] Lustre: server umount lustre-MDT0000 complete [ 1180.641044] LustreError: 72358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.648597] LustreError: 72358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1181.073120] LustreError: 65168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788824 with bad export cookie 335474618381761915 [ 1181.076499] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.078057] LustreError: 65168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1181.208926] Lustre: server umount lustre-MDT0001 complete [ 1183.531875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1184.756802] Lustre: DEBUG MARKER: files compared the same [ 1195.757438] Lustre: server umount lustre-OST0000 complete [ 1201.855670] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1202.978732] Key type lgssc unregistered [ 1203.120363] LNet: 74143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.124154] LNetError: 74143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.131478] LNet: Removed LNI 192.168.206.121@tcp [ 1203.458552] Key type .llcrypt unregistered [ 1203.460566] Key type ._llcrypt unregistered [ 1212.718069] Key type ._llcrypt registered [ 1212.719313] Key type .llcrypt registered [ 1212.760332] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1213.209780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1213.217275] alg: No test for adler32 (adler32-zlib) [ 1214.085573] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1214.190257] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1215.783145] Key type lgssc registered [ 1216.190056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.038734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.045938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.229055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.712827] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1224.812509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.940848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.441243] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1229.016334] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1230.895628] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.298041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1233.429036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.445381] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1238.039803] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.730506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1254.879588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1254.882027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.887730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.903820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.904929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1255.909881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.430481] Lustre: server umount lustre-OST0000 complete [ 1261.023983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1261.024154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.024509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.024514] Lustre: Skipped 1 previous similar message [ 1261.036896] Lustre: Skipped 1 previous similar message [ 1266.144040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.148396] Lustre: Skipped 2 previous similar messages [ 1271.263701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.266132] Lustre: Skipped 1 previous similar message [ 1273.311133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1273.377089] Lustre: server umount lustre-MDT0000 complete [ 1274.738511] LustreError: 75518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788917 with bad export cookie 11537097134806873867 [ 1274.740289] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.743331] LustreError: 75518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1274.857458] Lustre: server umount lustre-MDT0001 complete [ 1278.716456] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1279.813247] Key type lgssc unregistered [ 1279.952613] LNet: 79089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.955854] LNetError: 79089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.965331] LNet: Removed LNI 192.168.206.121@tcp [ 1280.299399] Key type .llcrypt unregistered [ 1280.300795] Key type ._llcrypt unregistered [ 1289.981253] Key type ._llcrypt registered [ 1289.982617] Key type .llcrypt registered [ 1290.023955] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1290.351972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.380460] alg: No test for adler32 (adler32-zlib) [ 1291.226747] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1291.320368] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1292.903144] Key type lgssc registered [ 1293.278747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1295.882035] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1295.890195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.327401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1316.448396] LustreError: 80455:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.121@tcp: failed processing log, type 4: rc = -110 [ 1343.071218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1345.072045] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1347.743264] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.559978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.607867] Lustre: *** cfs_fail_loc=706, val=0*** [ 1350.747115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.774799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1352.081887] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1355.176894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.326851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1356.687700] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1359.268790] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1361.270120] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1362.914945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.916279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.922663] Lustre: Skipped 1 previous similar message [ 1368.036200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.039603] Lustre: Skipped 2 previous similar messages [ 1368.205900] Lustre: server umount lustre-OST0000 complete [ 1370.592069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.594494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.599340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.153270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.153976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.630816] Lustre: server umount lustre-MDT0000 complete [ 1376.932111] LustreError: 83317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789019 with bad export cookie 6580791156763234864 [ 1376.935560] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.936758] LustreError: 83317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1377.051674] Lustre: server umount lustre-MDT0001 complete [ 1380.771744] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1381.893119] Key type lgssc unregistered [ 1382.049631] LNet: 84067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.053139] LNetError: 84067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.060440] LNet: Removed LNI 192.168.206.121@tcp [ 1382.392743] Key type .llcrypt unregistered [ 1382.394048] Key type ._llcrypt unregistered [ 1392.368123] Key type ._llcrypt registered [ 1392.369487] Key type .llcrypt registered [ 1392.422643] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1393.019123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.039253] alg: No test for adler32 (adler32-zlib) [ 1393.908915] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1394.007202] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1395.591111] Key type lgssc registered [ 1396.041522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1398.951252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1398.961864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.403933] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1404.362456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.560820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.079700] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1409.188216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.336401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.583557] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1414.434074] LustreError: 86147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.443064] LustreError: 86147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1414.753249] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1414.759180] Lustre: MGS: Client bb6c1709-b007-4094-b387-58b7958878d7 (at 0@lo) reconnecting [ 1414.770481] LustreError: 86161:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1414.854674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.886078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1416.447016] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1425.379609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1425.379831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.383818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1425.387077] Lustre: Skipped 1 previous similar message [ 1430.418641] Lustre: server umount lustre-OST0000 complete [ 1435.616693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.616780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.617357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.617360] Lustre: Skipped 1 previous similar message [ 1435.627219] Lustre: Skipped 1 previous similar message [ 1440.736066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.738814] Lustre: Skipped 2 previous similar messages [ 1445.855175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.856732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.861651] Lustre: Skipped 1 previous similar message [ 1445.901076] Lustre: server umount lustre-MDT0000 complete [ 1447.400391] Lustre: server umount lustre-MDT0001 complete [ 1448.929614] Lustre: server umount MGS complete [ 1452.975765] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:31:36 (1785789096) [ 1462.440468] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 1465.601862] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1476.647744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.891601] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1481.742132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.821241] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1481.836354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1481.872479] Lustre: lustre-MDT0001: new disk, initializing [ 1481.898265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.907046] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1481.910048] Lustre: 93303:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1481.913342] LustreError: 93303:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1483.209517] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1487.561751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1487.662643] Lustre: lustre-OST0000: new disk, initializing [ 1487.664683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1487.668079] Lustre: 94068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1487.692064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.721179] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1491.359772] Lustre: 94517:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000044b6391e x1872535525296000/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785789150 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1493.363112] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.580168] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.585323] LustreError: 84820:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7787ec8 x1872535525296768/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1495.593361] Lustre: MGS: Client 9f406c08-7223-420e-b42f-0b336867e967 (at 0@lo) reconnecting [ 1495.617263] Lustre: 95111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1495.625434] LustreError: 93289:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1495.638218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1495.673401] Lustre: lustre-MDT0000: new disk, initializing [ 1495.715923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.725347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1496.799654] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1496.799931] Lustre: 95140:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000055976e0f x1872535525303168/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785789155 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1496.810993] Lustre: 95140:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1497.104755] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1505.762022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1505.765378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1505.891385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1526.239836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.243095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.248201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.018291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.020601] Lustre: Skipped 6 previous similar messages [ 1540.063084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1540.119630] Lustre: server umount lustre-OST0000 complete [ 1543.138025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.143332] Lustre: Skipped 2 previous similar messages [ 1543.145030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1543.147458] LustreError: Skipped 1 previous similar message [ 1547.586673] Lustre: server umount lustre-MDT0000 complete [ 1548.257317] LustreError: 93275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.263964] LustreError: 93275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1563.103159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1563.229101] Lustre: server umount lustre-MDT0001 complete [ 1564.904810] Lustre: server umount MGS complete [ 1569.832674] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:33:33 (1785789213) [ 1577.199480] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1578.489589] Key type lgssc unregistered [ 1578.653334] LNet: 97640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.657133] LNetError: 97640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.666426] LNet: Removed LNI 192.168.206.121@tcp [ 1579.067552] Key type .llcrypt unregistered [ 1579.070610] Key type ._llcrypt unregistered [ 1588.096384] Key type ._llcrypt registered [ 1588.098178] Key type .llcrypt registered [ 1588.144402] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1588.631114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.640898] alg: No test for adler32 (adler32-zlib) [ 1589.529365] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1589.660030] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1591.255683] Key type lgssc registered [ 1591.755953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.133049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.145554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.345578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.173508] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1601.992916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.161750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.927894] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1607.113524] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1609.417670] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1612.061722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.221929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.445834] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1617.329252] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.480554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1626.598488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1626.599055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.602853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.613452] Lustre: Skipped 1 previous similar message [ 1626.618820] Lustre: Skipped 1 previous similar message [ 1627.803634] Lustre: server umount lustre-OST0000 complete [ 1631.471831] Lustre: server umount lustre-MDT0000 complete [ 1631.712280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.712792] LustreError: 98834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.717316] Lustre: Skipped 1 previous similar message [ 1631.728862] LustreError: 98834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1632.996256] LustreError: 98820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789276 with bad export cookie 17998316316032484251 [ 1633.000140] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.004240] LustreError: 98820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1633.154659] Lustre: server umount lustre-MDT0001 complete [ 1637.753156] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1638.928632] Key type lgssc unregistered [ 1639.069558] LNet: 102366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.073601] LNetError: 102366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.081659] LNet: Removed LNI 192.168.206.121@tcp [ 1639.451533] Key type .llcrypt unregistered [ 1639.452957] Key type ._llcrypt unregistered [ 1647.193861] Key type ._llcrypt registered [ 1647.195309] Key type .llcrypt registered [ 1647.234246] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1647.572137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.619234] alg: No test for adler32 (adler32-zlib) [ 1648.474068] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1648.562104] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1650.143150] Key type lgssc registered [ 1650.559230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1652.886187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1652.889668] LustreError: 103429:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1652.889819] Lustre: Skipped 1 previous similar message [ 1652.894879] LustreError: 103428:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1652.899085] LustreError: 103429:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1652.901738] LustreError: 103429:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1652.905613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.907086] LustreError: 103429:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1654.071586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.343470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.475449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1659.093820] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1661.089147] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing lsmod [ 1662.197361] LustreError: 104634:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1662.200452] LustreError: 104635:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1662.203407] LustreError: 104634:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1662.206643] LustreError: 104634:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1662.211833] LustreError: 104634:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1662.215062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.355782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.553069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.554664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.557963] Lustre: Skipped 2 previous similar messages [ 1667.562519] Lustre: Skipped 1 previous similar message [ 1669.597695] Lustre: server umount lustre-MDT0000 complete [ 1671.087397] LustreError: 103445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789314 with bad export cookie 9756642792576001661 [ 1671.088973] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.092886] LustreError: 103445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1671.607336] LustreError: 103898:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000601b4030 x1872535791615360/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1671.677625] Lustre: server umount lustre-MDT0001 complete [ 1683.411791] Lustre: server umount lustre-OST0000 complete [ 1686.543796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.703393] LustreError: 105697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.710813] LustreError: 105697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1686.728582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.123291] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1691.181706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.304078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.747620] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1695.451406] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.383164] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1699.708676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1699.859898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.036335] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1704.867514] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1711.142271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1717.217753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.221677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.226925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.312880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.313118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.314187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.315040] Lustre: Skipped 1 previous similar message [ 1726.433062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.435179] Lustre: Skipped 1 previous similar message [ 1731.039089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1731.088156] Lustre: server umount lustre-OST0000 complete [ 1732.575791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.578396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.582875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.672826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.655551] Lustre: server umount lustre-MDT0000 complete [ 1740.047768] LustreError: 105677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789383 with bad export cookie 9756642792576002347 [ 1740.050318] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.053128] LustreError: 105677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1740.169543] Lustre: server umount lustre-MDT0001 complete [ 1743.909466] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1745.020427] Key type lgssc unregistered [ 1745.169717] LNet: 109249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.174154] LNetError: 109249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.185502] LNet: Removed LNI 192.168.206.121@tcp [ 1745.536326] Key type .llcrypt unregistered [ 1745.537715] Key type ._llcrypt unregistered [ 1755.249436] Key type ._llcrypt registered [ 1755.250569] Key type .llcrypt registered [ 1755.297308] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1755.659533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1755.698050] alg: No test for adler32 (adler32-zlib) [ 1756.561784] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1756.661271] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1758.247230] Key type lgssc registered [ 1758.679475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.681929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1761.693044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.898163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.298781] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1767.127629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.245587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.613424] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1771.010604] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1772.746229] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.860860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.966108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.803837] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1779.135973] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1787.238736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1791.293488] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1792.020473] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1797.603090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.603299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1797.608306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.610151] LustreError: Skipped 1 previous similar message [ 1797.614767] Lustre: Skipped 1 previous similar message [ 1798.996382] Lustre: server umount lustre-OST0000 complete [ 1802.721144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.721768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.727611] Lustre: Skipped 2 previous similar messages [ 1806.571921] Lustre: server umount lustre-MDT0000 complete [ 1807.841702] LustreError: 110642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.848275] LustreError: 110642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1808.037973] LustreError: 113194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789451 with bad export cookie 5389149685218709065 [ 1808.041432] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.044057] LustreError: 113194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1808.154077] Lustre: server umount lustre-MDT0001 complete [ 1811.908676] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1812.971245] Key type lgssc unregistered [ 1813.108424] LNet: 114294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.111458] LNetError: 114294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.118484] LNet: Removed LNI 192.168.206.121@tcp [ 1813.432150] Key type .llcrypt unregistered [ 1813.433637] Key type ._llcrypt unregistered [ 1820.963314] Key type ._llcrypt registered [ 1820.964719] Key type .llcrypt registered [ 1821.006969] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1821.561573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.574983] alg: No test for adler32 (adler32-zlib) [ 1822.470437] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1822.567668] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1824.151216] Key type lgssc registered [ 1824.579843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.720283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.728065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.879716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.237531] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1833.095678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.210377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.574901] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1836.873408] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.600627] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1840.966371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.087872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.095783] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1845.494989] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.417387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1858.529584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.530577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.531262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.534694] Lustre: Skipped 1 previous similar message [ 1858.538026] LustreError: Skipped 1 previous similar message [ 1863.652800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.654989] Lustre: Skipped 2 previous similar messages [ 1864.596376] Lustre: server umount lustre-OST0000 complete [ 1868.769369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.769964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.773508] Lustre: Skipped 1 previous similar message [ 1872.152595] Lustre: server umount lustre-MDT0000 complete [ 1873.706634] LustreError: 116224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789516 with bad export cookie 5332873052677401575 [ 1873.710990] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.711224] LustreError: 116224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1873.835158] Lustre: server umount lustre-MDT0001 complete [ 1877.735784] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1878.912941] Key type lgssc unregistered [ 1879.049382] LNet: 119045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.052028] LNetError: 119045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.057281] LNet: Removed LNI 192.168.206.121@tcp [ 1879.390406] Key type .llcrypt unregistered [ 1879.392151] Key type ._llcrypt unregistered [ 1889.205193] Key type ._llcrypt registered [ 1889.206976] Key type .llcrypt registered [ 1889.247835] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1889.655921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1889.699125] alg: No test for adler32 (adler32-zlib) [ 1890.570762] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1890.678307] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1892.271744] Key type lgssc registered [ 1892.810885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1896.329506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1896.340933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.551245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.276867] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1902.798166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.964703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.654447] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1907.724759] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.879619] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1912.559630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1912.732304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.958326] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1917.939866] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1921.959518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1932.344579] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1934.050130] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1934.052953] Lustre: Skipped 2 previous similar messages [ 1936.080451] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1936.083316] Lustre: Skipped 2 previous similar messages [ 1937.933567] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1937.947418] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1946.804739] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1946.811048] Lustre: Skipped 2 previous similar messages [ 1952.772809] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.21@tcp 192.168.0.[2,10]@tcp [ 1955.156795] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.21/24@tcp in log lustre-MDT0000 [ 1955.161224] Lustre: Skipped 2 previous similar messages [ 1962.498801] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.21/24@tcp [ 1962.502102] Lustre: Skipped 1 previous similar message [ 1968.097308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.097550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.100422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.101076] LustreError: Skipped 1 previous similar message [ 1968.118433] Lustre: Skipped 1 previous similar message [ 1971.483062] Lustre: server umount lustre-OST0000 complete [ 1973.216575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.217212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.221820] Lustre: Skipped 1 previous similar message [ 1973.225733] Lustre: Skipped 2 previous similar messages [ 1978.336897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.339971] Lustre: Skipped 1 previous similar message [ 1979.301738] Lustre: server umount lustre-MDT0000 complete [ 1980.656314] LustreError: 120420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789623 with bad export cookie 11090065714628972998 [ 1980.660382] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.660653] LustreError: 120420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1980.771349] Lustre: server umount lustre-MDT0001 complete [ 1984.512935] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1985.751746] Key type lgssc unregistered [ 1985.882442] LNet: 125628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.885577] LNetError: 125628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.893423] LNet: Removed LNI 192.168.206.121@tcp [ 1986.283916] Key type .llcrypt unregistered [ 1986.285997] Key type ._llcrypt unregistered [ 1997.275996] Key type ._llcrypt registered [ 1997.276832] Key type .llcrypt registered [ 1997.317357] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1997.726912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.739571] alg: No test for adler32 (adler32-zlib) [ 1998.615732] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1998.738982] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2000.336057] Key type lgssc registered [ 2000.797566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.844912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2003.854637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.021424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.026097] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2005.036437] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.21/24@tcp [ 2006.548053] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2009.775973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.922464] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2009.934295] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.21/24@tcp [ 2009.955297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2011.533566] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2014.269589] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.203078] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2018.447794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.588306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.563056] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2022.966457] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2030.887183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2040.289183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.292747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.299034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2041.313218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2041.314689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.320609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.818591] Lustre: server umount lustre-OST0000 complete [ 2045.920028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2045.924832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.930872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.933190] Lustre: Skipped 1 previous similar message [ 2051.485432] Lustre: server umount lustre-MDT0000 complete [ 2051.553167] LustreError: 127165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.559589] LustreError: 127165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2052.912557] LustreError: 129012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789695 with bad export cookie 4789781956165418234 [ 2052.916172] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2052.916568] LustreError: 129012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2053.049349] Lustre: server umount lustre-MDT0001 complete [ 2057.046555] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2058.163529] Key type lgssc unregistered [ 2058.297626] LNet: 130772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.300731] LNetError: 130772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.310930] LNet: Removed LNI 192.168.206.121@tcp [ 2058.647440] Key type .llcrypt unregistered [ 2058.648530] Key type ._llcrypt unregistered [ 2068.264383] Key type ._llcrypt registered [ 2068.265871] Key type .llcrypt registered [ 2068.307119] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2074.591151] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2075.060263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.069937] alg: No test for adler32 (adler32-zlib) [ 2075.938061] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2076.039877] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2077.631195] Key type lgssc registered [ 2078.092250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2089.857319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2089.868332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.994577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2091.011933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2091.058413] Lustre: lustre-MDT0000: new disk, initializing [ 2091.090921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.119866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2092.762365] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2097.303033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.351066] Lustre: 134142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2097.379570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2097.382035] Lustre: Skipped 1 previous similar message [ 2097.422361] Lustre: lustre-MDT0001: new disk, initializing [ 2097.454892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.469213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2097.473070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2098.982297] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2102.836117] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.687208] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2107.037322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.135882] Lustre: lustre-OST0000: new disk, initializing [ 2107.137471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2107.139796] Lustre: 136110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2107.166360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.414256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2107.417598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2107.451766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2109.163092] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2113.213622] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2114.805750] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2114.892574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2116.552063] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2116.631588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2117.602605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.602765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.603595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2117.605082] LustreError: Skipped 1 previous similar message [ 2117.609995] Lustre: Skipped 1 previous similar message [ 2122.721812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.723780] Lustre: Skipped 2 previous similar messages [ 2123.498305] Lustre: server umount lustre-OST0000 complete [ 2127.841521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.842334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.844994] Lustre: Skipped 1 previous similar message [ 2131.158940] Lustre: server umount lustre-MDT0000 complete [ 2132.636427] LustreError: 134136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789775 with bad export cookie 10706140750837368569 [ 2132.639935] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.641313] LustreError: 134136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.768057] Lustre: server umount lustre-MDT0001 complete [ 2136.278195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.528918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.147548] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2141.261076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.437750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.104819] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2145.697555] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2147.570397] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2150.185956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2150.309778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.369922] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2154.974385] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2156.492635] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2161.657912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2163.062128] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2163.143899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2166.731788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2166.814394] Lustre: lustre-OST0001: new disk, initializing [ 2166.815890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2166.818347] Lustre: 141727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2166.849636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2168.769514] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2172.136462] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2173.965692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2173.969955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2174.012084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2174.270900] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2175.853309] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2175.963120] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2178.083302] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf488d03800.ost_server_uuid 50 [ 2178.626880] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cf488d03800.ost_server_uuid in FULL state after 0 sec [ 2184.167165] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.167553] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.170095] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2184.173575] Lustre: Skipped 1 previous similar message [ 2184.175764] Lustre: Skipped 1 previous similar message [ 2186.816712] Lustre: server umount lustre-OST0001 complete [ 2189.282531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.283294] LustreError: 140143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.284012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.286771] Lustre: Skipped 1 previous similar message [ 2189.293495] LustreError: 140143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2194.198657] Lustre: server umount lustre-OST0000 complete [ 2197.983939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2197.987461] LustreError: Skipped 1 previous similar message [ 2197.989436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.995434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.998079] Lustre: Skipped 1 previous similar message [ 2201.817935] Lustre: server umount lustre-MDT0000 complete [ 2203.005405] LustreError: 140165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789846 with bad export cookie 10706140750837369444 [ 2203.008700] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.010160] LustreError: 140165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2203.132423] Lustre: server umount lustre-MDT0001 complete [ 2209.178378] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2210.144816] Key type lgssc unregistered [ 2210.269445] LNet: 144738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.272612] LNetError: 144738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.279292] LNet: Removed LNI 192.168.206.121@tcp [ 2210.596407] Key type .llcrypt unregistered [ 2210.598050] Key type ._llcrypt unregistered [ 2223.266517] Key type ._llcrypt registered [ 2223.267801] Key type .llcrypt registered [ 2223.304737] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2229.550093] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2230.010168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.042746] alg: No test for adler32 (adler32-zlib) [ 2230.915461] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2231.020105] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2232.607115] Key type lgssc registered [ 2233.008269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2244.330613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2244.337320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.433579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2245.446812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2245.488695] Lustre: lustre-MDT0000: new disk, initializing [ 2245.525119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.537299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2247.086370] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2251.348194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.401580] Lustre: 148862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2251.426892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2251.429136] Lustre: Skipped 1 previous similar message [ 2251.470404] Lustre: lustre-MDT0001: new disk, initializing [ 2251.503161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.515500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2251.518945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2252.860379] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2256.585654] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2258.449091] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2260.721141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.834025] Lustre: lustre-OST0000: new disk, initializing [ 2260.835953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2260.839759] Lustre: 150830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2260.871105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.953190] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2266.105028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2266.108836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2266.142790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2266.672837] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.650728] Lustre: Failing over lustre-OST0000 [ 2268.725653] Lustre: server umount lustre-OST0000 complete [ 2281.721848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2281.890544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.901329] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2282.806551] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2284.160263] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2285.492976] Lustre: Failing over lustre-MDT0000 [ 2285.594548] Lustre: server umount lustre-MDT0000 complete [ 2287.073053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.077800] Lustre: Skipped 3 previous similar messages [ 2287.503142] Lustre: 147092:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785789914/real 1785789914] req@00000000b5ba6221 x1872536402945152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785789930 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2288.095193] Lustre: 147091:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785789914/real 1785789914] req@00000000aa0fc6f4 x1872536402946048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785789930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2292.191246] Lustre: 147091:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785789919/real 1785789919] req@000000009b154ea8 x1872536402947200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785789935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2292.192587] LustreError: 151752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.204066] LustreError: 151752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2292.715333] LustreError: 148869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.312655] LustreError: 149711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.319548] LustreError: 149711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2297.632087] Lustre: 147092:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785789924/real 1785789924] req@000000007dc4f607 x1872536402948352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785789940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2298.623493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.689739] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.695684] LustreError: 153047:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.121@tcp: namespace resource [0x65727473756c:0x2:0x0].0x0 (0000000033d1dd22) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2298.701918] Lustre: 148865:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2298.793304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2298.814331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.836473] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2300.373902] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2302.955784] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2303.975762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2303.984582] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2303.989899] Lustre: Skipped 1 previous similar message [ 2304.006874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2309.092896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.093756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.096051] LustreError: Skipped 1 previous similar message [ 2309.100165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.102992] Lustre: Skipped 1 previous similar message [ 2311.184122] Lustre: server umount lustre-OST0000 complete [ 2314.208473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2314.209365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.212402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.216394] Lustre: Skipped 2 previous similar messages [ 2314.221460] Lustre: Skipped 1 previous similar message [ 2318.619493] Lustre: server umount lustre-MDT0000 complete [ 2319.327919] LustreError: 148873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.332317] LustreError: 148873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2319.871289] LustreError: 148853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789962 with bad export cookie 9662809128885000877 [ 2319.874933] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.875702] LustreError: 148853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2320.005335] Lustre: server umount lustre-MDT0001 complete [ 2323.600123] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2324.619270] Key type lgssc unregistered [ 2324.742347] LNet: 154477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.745100] LNetError: 154477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.754304] LNet: Removed LNI 192.168.206.121@tcp [ 2325.066427] Key type .llcrypt unregistered [ 2325.067805] Key type ._llcrypt unregistered [ 2345.505967] Key type ._llcrypt registered [ 2345.507232] Key type .llcrypt registered [ 2345.548460] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2346.120854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.159601] alg: No test for adler32 (adler32-zlib) [ 2347.036716] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2347.141281] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2348.735200] Key type lgssc registered [ 2349.181359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2351.924466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2351.932581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.105403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.710625] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2358.535316] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.547324] LustreError: 156019:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2360.552211] LustreError: 156019:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2360.556085] LustreError: 156019:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2360.688692] Lustre: server umount lustre-MDT0000 complete [ 2366.122323] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2369.493551] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2381.826349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.909825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2381.919180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2381.955281] Lustre: lustre-MDT0000: new disk, initializing [ 2381.980484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.987376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2383.381769] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2387.836178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.885814] Lustre: 159316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2387.919733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2387.924763] Lustre: Skipped 1 previous similar message [ 2387.927414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2387.971101] Lustre: lustre-MDT0001: new disk, initializing [ 2388.032515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2388.054358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2388.062113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2389.777975] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2394.161700] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2396.237252] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.589177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.690992] Lustre: lustre-OST0000: new disk, initializing [ 2398.693069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2398.696131] Lustre: 161284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2398.724663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.799924] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2403.330052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2403.336190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2403.381667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2404.763058] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2406.648674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2407.360551] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:47:30 (1785790050) [ 2480.095949] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.095970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.096198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.100143] Lustre: Skipped 1 previous similar message [ 2482.397123] Lustre: server umount lustre-MDT0000 complete [ 2483.553455] LustreError: 159308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790126 with bad export cookie 17000104106865014468 [ 2483.554338] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.556653] LustreError: 159308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.215365] LustreError: 162207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.215446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.219666] LustreError: 162207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2485.223515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2485.227656] Lustre: Skipped 2 previous similar messages [ 2488.223509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2488.223509] LustreError: 159322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.696230] Lustre: server umount lustre-MDT0001 complete [ 2497.159070] Lustre: server umount lustre-OST0000 complete [ 2500.222560] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2503.479148] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2515.395051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.484731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2515.497695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2515.543760] Lustre: lustre-MDT0000: new disk, initializing [ 2515.568568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.575741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2516.997234] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2521.106074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.149673] Lustre: 165909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2521.174356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2521.176872] Lustre: Skipped 1 previous similar message [ 2521.214748] Lustre: lustre-MDT0001: new disk, initializing [ 2521.259651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2521.263343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2522.568607] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2526.089136] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.790612] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2529.895210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.995743] Lustre: lustre-OST0000: new disk, initializing [ 2529.997758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2530.000778] Lustre: 167874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2530.029698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.032182] Lustre: Skipped 1 previous similar message [ 2531.894863] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2535.517555] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2536.951054] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2540.024816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2540.028496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2540.064922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2541.108207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2542.479707] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2542.548434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2545.124031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.124951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2545.129349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.134398] Lustre: Skipped 1 previous similar message [ 2549.329377] Lustre: server umount lustre-OST0000 complete [ 2552.287814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2552.289813] LustreError: Skipped 1 previous similar message [ 2552.291218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.295756] Lustre: Skipped 1 previous similar message [ 2555.359947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.361832] Lustre: Skipped 2 previous similar messages [ 2556.692121] Lustre: server umount lustre-MDT0000 complete [ 2557.835098] LustreError: 166703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790200 with bad export cookie 17000104106865173298 [ 2557.837744] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.839070] LustreError: 166703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2557.935593] Lustre: server umount lustre-MDT0001 complete [ 2560.491828] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:50:03 (1785790203) [ 2565.402973] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2568.266449] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2579.330774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.426772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2579.438478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2579.474713] Lustre: lustre-MDT0000: new disk, initializing [ 2579.495297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.501100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2580.846653] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2584.840532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.879255] Lustre: 172656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2584.968791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2586.231480] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2589.773269] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.478509] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2593.653078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.754256] Lustre: 174636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2595.513124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2595.548071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2595.655694] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2599.103874] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2600.929503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2600.929631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.932221] LustreError: Skipped 1 previous similar message [ 2600.935684] Lustre: Skipped 1 previous similar message [ 2600.936131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.940343] Lustre: Skipped 1 previous similar message [ 2606.930661] Lustre: server umount lustre-OST0000 complete [ 2610.655825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2610.658838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.663878] Lustre: Skipped 1 previous similar message [ 2614.360272] Lustre: server umount lustre-MDT0000 complete [ 2615.447040] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790258 with bad export cookie 17000104106865174166 [ 2615.450316] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2615.450425] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.559667] Lustre: server umount lustre-MDT0001 complete [ 2618.073252] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:51:01 (1785790261) [ 2622.967376] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2625.805200] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2636.663489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.733122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2636.743145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2636.747077] Lustre: Skipped 3 previous similar messages [ 2636.778689] Lustre: lustre-MDT0000: new disk, initializing [ 2636.780272] Lustre: Skipped 2 previous similar messages [ 2636.798370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.800174] Lustre: Skipped 2 previous similar messages [ 2636.804562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2636.807750] Lustre: Skipped 2 previous similar messages [ 2638.066825] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2641.929978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.976254] Lustre: 178950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2642.077033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2643.371132] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2647.128792] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2648.871630] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2650.836887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.929958] Lustre: 180924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2652.761332] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2654.939950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2656.503685] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2660.320730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.322488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.323829] LustreError: Skipped 1 previous similar message [ 2660.328212] Lustre: Skipped 1 previous similar message [ 2660.331187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.335042] Lustre: Skipped 6 previous similar messages [ 2664.466589] Lustre: server umount lustre-OST0000 complete [ 2680.287109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2680.348674] Lustre: server umount lustre-MDT0000 complete [ 2680.800318] LustreError: 178960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.807918] LustreError: 178960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2681.575434] LustreError: 179749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790324 with bad export cookie 17000104106865175237 [ 2681.579268] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.581152] LustreError: 179749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2681.705809] Lustre: server umount lustre-MDT0001 complete [ 2685.251027] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2686.182098] Key type lgssc unregistered [ 2686.304363] LNet: 182780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.306810] LNetError: 182780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.315254] LNet: Removed LNI 192.168.206.121@tcp [ 2686.583403] Key type .llcrypt unregistered [ 2686.584254] Key type ._llcrypt unregistered [ 2695.054036] Key type ._llcrypt registered [ 2695.055099] Key type .llcrypt registered [ 2695.085224] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2695.340943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.351590] alg: No test for adler32 (adler32-zlib) [ 2696.209382] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2696.301087] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2697.871190] Key type lgssc registered [ 2698.180348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.747333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2700.753682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.909465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.257024] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2706.024492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.148737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2707.507376] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2709.900144] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2711.793756] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2714.017796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.140764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.140912] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2718.548144] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2726.436479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2741.731430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.731651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.735384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.738783] Lustre: Skipped 1 previous similar message [ 2741.742270] Lustre: Skipped 1 previous similar message [ 2746.576877] Lustre: server umount lustre-OST0000 complete [ 2751.969829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.970641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.974516] Lustre: Skipped 1 previous similar message [ 2751.977824] Lustre: Skipped 1 previous similar message [ 2757.088182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.091090] Lustre: Skipped 1 previous similar message [ 2761.184196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.186287] Lustre: Skipped 1 previous similar message [ 2762.207124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2762.272608] Lustre: server umount lustre-MDT0000 complete [ 2763.460144] LustreError: 184153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790406 with bad export cookie 905538535728539954 [ 2763.463505] LustreError: 184153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2763.464042] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.577741] Lustre: server umount lustre-MDT0001 complete [ 2767.176469] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2768.211390] Key type lgssc unregistered [ 2768.346361] LNet: 187726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2768.349837] LNetError: 187726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2768.359409] LNet: Removed LNI 192.168.206.121@tcp [ 2768.684440] Key type .llcrypt unregistered [ 2768.685489] Key type ._llcrypt unregistered [ 2777.609708] Key type ._llcrypt registered [ 2777.610708] Key type .llcrypt registered [ 2777.645697] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2778.045948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2778.052993] alg: No test for adler32 (adler32-zlib) [ 2778.893150] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2778.975708] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2780.551136] Key type lgssc registered [ 2780.885278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2783.230922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2783.236048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.358709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.597244] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2788.269350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.381087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2789.592983] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2791.827076] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2793.608653] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2795.742454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.859707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.707410] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2800.101858] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2809.190944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2814.434713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.435192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.436081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.445549] Lustre: Skipped 1 previous similar message [ 2815.987061] Lustre: lustre-OST0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 2815.989690] Lustre: Skipped 1 previous similar message [ 2819.218350] Lustre: server umount lustre-OST0000 complete [ 2821.589932] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9cf48958f000.ost_server_uuid 50 [ 2822.145189] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2832.131198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.132180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.137487] Lustre: Skipped 1 previous similar message [ 2832.140533] Lustre: Skipped 1 previous similar message [ 2834.208462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.211322] Lustre: Skipped 1 previous similar message [ 2838.295498] Lustre: server umount lustre-MDT0000 complete [ 2839.521102] LustreError: 189119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.528361] LustreError: 189119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2839.540477] LustreError: 190962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790482 with bad export cookie 17789310667463235643 [ 2839.543788] LustreError: 190962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2839.543863] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.660917] Lustre: server umount lustre-MDT0001 complete [ 2842.458070] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:54:45 (1785790485) [ 2844.716474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.880391] LustreError: 193031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.907675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.190377] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2848.828882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.943863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.184032] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2852.381918] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2854.058693] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2856.119469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.937612] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2860.183594] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2862.266052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.343613] Lustre: lustre-OST0001: new disk, initializing [ 2862.345446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2862.347615] Lustre: 195865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2862.379262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.381748] Lustre: Skipped 1 previous similar message [ 2864.298120] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2867.202297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2867.205517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2867.237339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2867.977113] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2869.856463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.860740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.939907] Lustre: server umount lustre-OST0000 complete [ 2875.364077] LustreError: 194848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.369696] LustreError: 194848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2876.776488] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2876.850365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2878.059227] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2878.130849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2878.444908] LustreError: 196506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.564111] LustreError: 196674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.569573] LustreError: 196674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2890.721664] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.722518] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2890.725293] Lustre: Skipped 1 previous similar message [ 2890.726700] Lustre: Skipped 2 previous similar messages [ 2894.160909] Lustre: server umount lustre-OST0001 complete [ 2895.840820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.448043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.450911] LustreError: Skipped 1 previous similar message [ 2901.528701] Lustre: server umount lustre-MDT0000 complete [ 2902.808128] LustreError: 193013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790545 with bad export cookie 17789310667463236700 [ 2902.811462] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.811466] LustreError: 193013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2902.932619] Lustre: server umount lustre-MDT0001 complete [ 2910.754703] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:55:54 (1785790554) [ 2912.921795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.020426] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2913.031865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2913.104568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.392134] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2917.251307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.282722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2917.295482] Lustre: 199276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2918.769738] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2921.192974] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2923.070881] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2925.328906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2925.397965] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2927.436233] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2929.881946] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2932.156728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.184899] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2933.032257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2933.034898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2934.250466] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2936.634403] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.342696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.343883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.345411] LustreError: Skipped 1 previous similar message [ 2938.351813] Lustre: Skipped 1 previous similar message [ 2938.352608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.359013] Lustre: Skipped 4 previous similar messages [ 2939.584068] Lustre: server umount lustre-OST0000 complete [ 2941.941374] LustreError: 201129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.949142] LustreError: 201129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2953.696593] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.234965] Lustre: server umount lustre-OST0001 complete [ 2958.303802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2958.306450] LustreError: Skipped 1 previous similar message [ 2958.307995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.312064] Lustre: Skipped 3 previous similar messages [ 2963.541962] Lustre: server umount lustre-MDT0000 complete [ 2963.935917] LustreError: 200040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.941382] LustreError: 200040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2964.698229] LustreError: 200025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790607 with bad export cookie 17789310667463237862 [ 2964.700308] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.702244] LustreError: 200025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2964.816946] Lustre: server umount lustre-MDT0001 complete [ 2972.057658] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:56:55 (1785790615) [ 2977.036049] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2979.918775] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2990.967417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.040496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2991.049983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2991.086632] Lustre: lustre-MDT0000: new disk, initializing [ 2991.106949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.109792] Lustre: Skipped 3 previous similar messages [ 2991.115167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2992.341682] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2996.254700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.291199] Lustre: 207332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2996.312650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2996.314144] Lustre: Skipped 1 previous similar message [ 2996.348623] Lustre: lustre-MDT0001: new disk, initializing [ 2996.385676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2996.389126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2997.660621] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3001.253694] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3002.949113] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3005.065335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.167254] Lustre: lustre-OST0000: new disk, initializing [ 3005.169449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3005.171481] Lustre: 209299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3006.918745] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3010.406423] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.661627] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3014.649868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3014.653338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3014.688701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3014.781979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3015.994739] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3016.056221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3019.745368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3019.747659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.749772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.751070] Lustre: Skipped 2 previous similar messages [ 3019.753305] Lustre: Skipped 4 previous similar messages [ 3022.735473] Lustre: server umount lustre-OST0000 complete [ 3030.175872] Lustre: server umount lustre-MDT0000 complete [ 3031.393048] LustreError: 207325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790674 with bad export cookie 17789310667463239528 [ 3031.395544] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.399573] LustreError: 207325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3031.500464] Lustre: server umount lustre-MDT0001 complete [ 3034.169857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.330583] LustreError: 211537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.676727] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3038.380141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.826722] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3042.192933] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3043.951565] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3046.079644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3048.099092] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3050.491714] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3051.813724] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3060.026740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3061.350030] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3061.414415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3063.775773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.778427] LustreError: Skipped 1 previous similar message [ 3068.175476] Lustre: server umount lustre-OST0000 complete [ 3070.136842] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3071.433966] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3094.696681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.795612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.797673] Lustre: Skipped 5 previous similar messages [ 3096.518169] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3098.764290] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3099.235689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3104.224978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3104.226842] Lustre: Skipped 8 previous similar messages [ 3106.251118] Lustre: server umount lustre-OST0000 complete [ 3109.344915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.349391] Lustre: Skipped 5 previous similar messages [ 3113.622651] Lustre: server umount lustre-MDT0000 complete [ 3114.464307] LustreError: 213743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.470264] LustreError: 213743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3114.727882] LustreError: 211518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790757 with bad export cookie 17789310667463240396 [ 3114.730778] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.731667] LustreError: 211518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.831797] Lustre: server umount lustre-MDT0001 complete [ 3117.289945] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:59:20 (1785790760) [ 3119.297866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.657683] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3123.239867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.583971] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3126.923769] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3128.695352] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3130.908505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.006130] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3135.531969] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3136.977963] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3145.232069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3146.715513] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3146.799912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3149.163820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.246276] Lustre: lustre-OST0001: new disk, initializing [ 3149.247846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3149.250354] Lustre: 220971:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3151.207182] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3154.883465] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.968995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3155.972556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3156.005941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3156.322642] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3156.394552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3157.754832] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3157.831743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3161.058543] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.061700] LustreError: Skipped 2 previous similar messages [ 3164.546243] Lustre: server umount lustre-OST0001 complete [ 3166.465590] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3166.542259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3167.858932] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3167.924234] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3191.360876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.342700] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3195.707116] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3196.205039] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3196.900514] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3196.903908] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3196.903935] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3204.029896] Lustre: server umount lustre-OST0001 complete [ 3222.090798] Lustre: server umount lustre-OST0000 complete [ 3226.079673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.081903] LustreError: Skipped 3 previous similar messages [ 3229.526199] Lustre: server umount lustre-MDT0000 complete [ 3230.645512] LustreError: 220970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790873 with bad export cookie 17789310667463241327 [ 3230.648404] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.649116] LustreError: 220970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.839459] Lustre: server umount lustre-MDT0001 complete [ 3243.893083] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:01:27 (1785790887) [ 3245.955550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.043942] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3246.052626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3246.114603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.116715] Lustre: Skipped 5 previous similar messages [ 3247.389854] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3249.994952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.026876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3250.036577] Lustre: 226216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3251.377497] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3253.725660] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.516111] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3257.728564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.795166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3259.835496] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3262.245764] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3271.090172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.112048] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3273.027556] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3275.353637] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3275.854061] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3277.252561] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3277.326730] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3278.663347] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3278.734400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3280.481675] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf4840cd000.ost_server_uuid 50 [ 3280.983184] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cf4840cd000.ost_server_uuid in FULL state after 0 sec [ 3281.322615] Lustre: Permanently deactivating lustre-OST0001 [ 3281.328192] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3282.127507] Lustre: Permanently reactivating lustre-OST0001 [ 3284.964660] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.967928] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3284.969182] Lustre: Skipped 9 previous similar messages [ 3284.971533] Lustre: Skipped 13 previous similar messages [ 3289.154110] Lustre: server umount lustre-OST0001 complete [ 3290.085423] LustreError: 228058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.092605] LustreError: 228058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 3296.526519] Lustre: server umount lustre-OST0000 complete [ 3312.095099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3312.154748] Lustre: server umount lustre-MDT0000 complete [ 3313.336224] LustreError: 226207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790956 with bad export cookie 17789310667463242601 [ 3313.338247] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.340715] LustreError: 226207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3313.468050] Lustre: server umount lustre-MDT0001 complete [ 3320.867683] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:02:44 (1785790964) [ 3323.262452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.350803] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3323.361667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3323.365513] Lustre: Skipped 5 previous similar messages [ 3324.655820] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3327.264633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.297315] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3327.307939] Lustre: 232898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3328.678721] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3330.950618] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.630764] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3334.780162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3334.850960] Lustre: Permanently deactivating lustre-OST0000 [ 3334.856053] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3336.703630] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3338.033268] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3338.901247] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3341.009023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.933120] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3344.027632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3345.274168] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.774701] Lustre: Permanently reactivating lustre-OST0000 [ 3354.101375] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3354.122109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3359.715108] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.718662] LustreError: Skipped 5 previous similar messages [ 3363.968110] Lustre: server umount lustre-OST0001 complete [ 3371.273895] Lustre: server umount lustre-OST0000 complete [ 3378.648820] Lustre: server umount lustre-MDT0000 complete [ 3379.816102] LustreError: 233639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791022 with bad export cookie 17789310667463244239 [ 3379.818572] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.820163] LustreError: 233639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3379.936654] Lustre: server umount lustre-MDT0001 complete [ 3383.514705] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3384.433363] Key type lgssc unregistered [ 3384.549299] LNet: 237768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.551341] LNetError: 237768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.561299] LNet: Removed LNI 192.168.206.121@tcp [ 3384.858566] Key type .llcrypt unregistered [ 3384.860211] Key type ._llcrypt unregistered [ 3393.197235] Key type ._llcrypt registered [ 3393.198351] Key type .llcrypt registered [ 3393.239117] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3393.686817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3393.699869] alg: No test for adler32 (adler32-zlib) [ 3394.563570] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3394.667524] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3396.247129] Key type lgssc registered [ 3396.646401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.948650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3399.955303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.126718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.533229] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3405.297882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.341871] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3405.353284] Lustre: 239201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3405.358099] Lustre: Permanently deactivating lustre-MDT0001 [ 3405.363507] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3405.475523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3406.834152] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3409.274931] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3410.939089] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3411.050833] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3413.187024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.303081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.083859] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3415.329627] LustreError: 241052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.333679] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3415.335808] LustreError: 241052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3417.386453] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3419.502911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.582081] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.403837] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3423.592299] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.741448] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3426.792967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3426.793176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3430.292655] Lustre: Permanently reactivating lustre-MDT0001 [ 3430.298107] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3430.300617] Lustre: Skipped 2 previous similar messages [ 3437.058555] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.063729] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3437.066624] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3437.070160] LustreError: 242775:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3437.070578] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3437.073078] LustreError: 242775:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3437.078292] LustreError: 242775:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.121@tcp: cfg command failed: rc = -108 [ 3437.081300] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3437.081300] [ 3437.084277] LustreError: 239203:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3437.552832] Lustre: lustre-MDT0001: Client 3b042091-e5d4-4c43-b70b-5c90590ade32 (at 192.168.206.21@tcp) reconnecting [ 3440.279705] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3440.781185] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3442.490224] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3442.565945] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3443.879430] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3443.951251] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3444.762473] Lustre: Permanently deactivating lustre-MDT0001 [ 3444.767707] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3444.769608] Lustre: Skipped 2 previous similar messages [ 3452.924283] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3462.111784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3462.113647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.117874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.626467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.628332] Lustre: Skipped 2 previous similar messages [ 3463.830037] Lustre: server umount lustre-MDT0000 complete [ 3464.927196] LustreError: 242754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791107 with bad export cookie 2447058067944985780 [ 3464.930164] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.931392] LustreError: 242754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3465.024617] Lustre: server umount lustre-MDT0001 complete [ 3472.248369] Lustre: server umount lustre-OST0000 complete [ 3476.053281] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:05:19 (1785791119) [ 3479.649398] Lustre: server umount lustre-OST0001 complete [ 3481.548318] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 3484.647984] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3495.674653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.754199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3495.757761] Lustre: Skipped 2 previous similar messages [ 3495.765596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3495.801127] Lustre: lustre-MDT0000: new disk, initializing [ 3495.821458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.827187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3497.126807] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3501.133692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.179721] Lustre: 247553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3501.201330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3501.203177] Lustre: Skipped 1 previous similar message [ 3501.234588] Lustre: lustre-MDT0001: new disk, initializing [ 3501.273211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3501.276123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3502.533054] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3506.143145] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.811939] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3510.087549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.194983] Lustre: lustre-OST0000: new disk, initializing [ 3510.196386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3510.198581] Lustre: 249518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3510.225252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.227544] Lustre: Skipped 1 previous similar message [ 3511.983388] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3512.441058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3512.444438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3512.473481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3515.560611] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3516.163502] LustreError: 247560:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3520.195995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.254869] Lustre: lustre-OST0001: new disk, initializing [ 3520.256270] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3520.258552] Lustre: 250635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3522.142789] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3522.238714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3522.241543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3522.273183] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3525.726539] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3526.210346] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3526.256046] LustreError: 247560:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3526.264265] LustreError: 248777:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3536.351076] LustreError: 248777:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3537.888839] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.890643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.890904] LustreError: Skipped 1 previous similar message [ 3537.894504] Lustre: Skipped 3 previous similar messages [ 3537.895129] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3537.899602] Lustre: Skipped 1 previous similar message [ 3541.491406] Lustre: lustre-OST0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 3543.226141] Lustre: server umount lustre-OST0001 complete [ 3548.131335] LustreError: 249502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.136416] LustreError: 249502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3553.251639] LustreError: 249501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.256270] LustreError: 249501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3558.370791] LustreError: 249958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.370979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.371804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3558.371809] Lustre: Skipped 2 previous similar messages [ 3558.376835] LustreError: 249958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3558.382149] Lustre: Skipped 2 previous similar messages [ 3561.295497] Lustre: server umount lustre-OST0000 complete [ 3562.975738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3562.978338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.608246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.610148] Lustre: Skipped 5 previous similar messages [ 3568.733167] Lustre: server umount lustre-MDT0000 complete [ 3569.927457] LustreError: 248346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791212 with bad export cookie 2447058067944988559 [ 3569.929798] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.931755] LustreError: 248346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3576.108959] Lustre: server umount lustre-MDT0001 complete [ 3579.519567] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3580.522141] Key type lgssc unregistered [ 3580.642396] LNet: 252683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.645337] LNetError: 252683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3580.653387] LNet: Removed LNI 192.168.206.121@tcp [ 3580.949987] Key type .llcrypt unregistered [ 3580.951218] Key type ._llcrypt unregistered [ 3594.597422] Key type ._llcrypt registered [ 3594.598943] Key type .llcrypt registered [ 3594.634063] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3595.013971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3595.039091] alg: No test for adler32 (adler32-zlib) [ 3595.893984] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3595.978364] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3597.551157] Key type lgssc registered [ 3597.884995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3600.356109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3600.361823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.442176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3601.447857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3601.495633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.772125] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3605.480548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.521920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3605.535585] Lustre: 254865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3605.627465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3606.886134] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3609.146350] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3610.903207] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3613.042501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3613.108623] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3613.167207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.054546] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3617.469572] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3621.389646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3662.305886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.306971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.309694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.311822] LustreError: Skipped 1 previous similar message [ 3662.313264] Lustre: Skipped 1 previous similar message [ 3667.426110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.427813] Lustre: Skipped 1 previous similar message [ 3667.541203] Lustre: server umount lustre-OST0000 complete [ 3668.919687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3672.836584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.938112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.747706] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3677.034377] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3692.516706] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3692.522943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3692.524909] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3692.531403] Lustre: Skipped 1 previous similar message [ 3712.995157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.995739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.998237] LustreError: Skipped 1 previous similar message [ 3712.998877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.001837] Lustre: Skipped 2 previous similar messages [ 3716.248785] Lustre: server umount lustre-OST0000 complete [ 3718.112674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.113239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.118094] Lustre: Skipped 1 previous similar message [ 3718.120953] Lustre: Skipped 2 previous similar messages [ 3723.232077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.234702] Lustre: Skipped 1 previous similar message [ 3723.537929] Lustre: server umount lustre-MDT0000 complete [ 3724.658444] LustreError: 254859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791367 with bad export cookie 6822360562156773360 [ 3724.661576] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.662334] LustreError: 254859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3724.789546] Lustre: server umount lustre-MDT0001 complete [ 3728.294554] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3729.264830] Key type lgssc unregistered [ 3729.384394] LNet: 260277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3729.387439] LNetError: 260277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3729.400317] LNet: Removed LNI 192.168.206.121@tcp [ 3729.677556] Key type .llcrypt unregistered [ 3729.678979] Key type ._llcrypt unregistered [ 3738.443491] Key type ._llcrypt registered [ 3738.444534] Key type .llcrypt registered [ 3738.485083] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3738.788370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3738.843078] alg: No test for adler32 (adler32-zlib) [ 3739.694818] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3739.786845] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3741.367105] Key type lgssc registered [ 3741.700403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3744.102150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3744.107548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.262987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.556862] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3749.176869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.282629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.514458] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3752.759182] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3754.541597] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3756.663236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.769774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.556043] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3760.869079] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3770.085561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3780.577304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.577710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.578663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.581201] Lustre: Skipped 1 previous similar message [ 3780.586072] LustreError: Skipped 1 previous similar message [ 3784.455411] Lustre: server umount lustre-OST0000 complete [ 3785.696072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.701573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.703411] Lustre: Skipped 1 previous similar message [ 3790.303772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3790.305768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.309704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.767544] Lustre: server umount lustre-MDT0000 complete [ 3792.942184] LustreError: 261657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791436 with bad export cookie 12387514650205899952 [ 3792.944563] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.945644] LustreError: 261657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3793.050330] Lustre: server umount lustre-MDT0001 complete [ 3796.522256] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3797.543075] Key type lgssc unregistered [ 3797.683426] LNet: 265998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3797.686241] LNetError: 265998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3797.694618] LNet: Removed LNI 192.168.206.121@tcp [ 3798.011685] Key type .llcrypt unregistered [ 3798.012956] Key type ._llcrypt unregistered [ 3805.063867] Key type ._llcrypt registered [ 3805.066135] Key type .llcrypt registered [ 3805.114635] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3805.653898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3805.680781] alg: No test for adler32 (adler32-zlib) [ 3806.536504] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3806.627856] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3808.215143] Key type lgssc registered [ 3808.560294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3811.085399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3811.091179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.226957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.528482] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3816.255188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.370290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.699621] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3819.987520] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3821.827091] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.971222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.091813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3825.939610] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3828.191555] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3837.417341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3842.528909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.529105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.529923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.533957] Lustre: Skipped 1 previous similar message [ 3844.428526] Lustre: server umount lustre-OST0000 complete [ 3847.135861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3847.138204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.143131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3847.144748] Lustre: Skipped 1 previous similar message [ 3851.724932] Lustre: server umount lustre-MDT0000 complete [ 3852.768198] LustreError: 267197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.774947] LustreError: 267197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3852.937956] LustreError: 267177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791495 with bad export cookie 17446455926307527804 [ 3852.940952] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.941878] LustreError: 267177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3853.054342] Lustre: server umount lustre-MDT0001 complete [ 3856.484995] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3857.508875] Key type lgssc unregistered [ 3857.635315] LNet: 270890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.637547] LNetError: 270890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.644264] LNet: Removed LNI 192.168.206.121@tcp [ 3857.947807] Key type .llcrypt unregistered [ 3857.948959] Key type ._llcrypt unregistered [ 3866.643263] Key type ._llcrypt registered [ 3866.644502] Key type .llcrypt registered [ 3866.685132] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3867.028659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3867.041801] alg: No test for adler32 (adler32-zlib) [ 3867.890378] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3867.970900] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3869.551100] Key type lgssc registered [ 3869.903780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3872.420110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3872.427164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.574057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.800722] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3877.411727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.528860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3878.869638] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3881.143301] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3882.988241] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3885.252488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3885.364460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.300255] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3889.708568] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3899.691646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3908.063841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.066648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.072366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3910.112201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3910.114294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.114809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.674923] Lustre: server umount lustre-OST0000 complete [ 3915.234554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.235659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.238832] Lustre: Skipped 1 previous similar message [ 3915.243924] Lustre: Skipped 2 previous similar messages [ 3920.352049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3920.354068] Lustre: Skipped 1 previous similar message [ 3920.984431] Lustre: server umount lustre-MDT0000 complete [ 3922.190361] LustreError: 272262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791565 with bad export cookie 5609013897829484507 [ 3922.193501] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.194218] LustreError: 272262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3922.305801] Lustre: server umount lustre-MDT0001 complete [ 3925.954415] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3926.987277] Key type lgssc unregistered [ 3927.115358] LNet: 276605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3927.117580] LNetError: 276605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3927.129339] LNet: Removed LNI 192.168.206.121@tcp [ 3927.444362] Key type .llcrypt unregistered [ 3927.445561] Key type ._llcrypt unregistered [ 3934.260728] Key type ._llcrypt registered [ 3934.261847] Key type .llcrypt registered [ 3934.299066] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3934.597500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3934.603736] alg: No test for adler32 (adler32-zlib) [ 3935.468452] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3935.569312] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3937.151145] Key type lgssc registered [ 3937.543546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3940.124103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3940.130069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.284804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.610206] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3945.357971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.478131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3946.808408] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3949.090087] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3950.798265] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3952.925167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3953.025971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.715706] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3956.839274] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.374881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3971.553275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.553441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.561603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.564373] Lustre: Skipped 1 previous similar message [ 3973.068714] Lustre: server umount lustre-OST0000 complete [ 3976.160323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3976.164046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.168105] Lustre: Skipped 1 previous similar message [ 3976.169878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3976.399627] Lustre: server umount lustre-MDT0000 complete [ 3976.672516] LustreError: 277800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.677373] LustreError: 277800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3977.631092] LustreError: 277866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791620 with bad export cookie 8960852194878662001 [ 3977.633860] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3977.636255] LustreError: 277866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3977.763732] Lustre: server umount lustre-MDT0001 complete [ 3981.245521] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3982.248746] Key type lgssc unregistered [ 3982.385305] LNet: 281532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3982.387712] LNetError: 281532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3982.396254] LNet: Removed LNI 192.168.206.121@tcp [ 3982.688955] Key type .llcrypt unregistered [ 3982.689824] Key type ._llcrypt unregistered [ 3986.069433] Key type ._llcrypt registered [ 3986.070311] Key type .llcrypt registered [ 3986.105956] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3991.337626] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 3997.612462] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3997.989796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3998.023074] alg: No test for adler32 (adler32-zlib) [ 3998.877978] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3998.972364] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4000.559167] Key type lgssc registered [ 4000.943508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4012.392838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4012.397635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.495390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4013.505511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4013.546698] Lustre: lustre-MDT0000: new disk, initializing [ 4013.568147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.576168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4014.949300] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4019.027571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.070907] Lustre: 285056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4019.098362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4019.101113] Lustre: Skipped 1 previous similar message [ 4019.138911] Lustre: lustre-MDT0001: new disk, initializing [ 4019.174041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.188085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4019.191958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4020.608041] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4024.324509] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4026.209691] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4028.357797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4028.447165] Lustre: lustre-OST0000: new disk, initializing [ 4028.448796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4028.451035] Lustre: 287024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4028.478266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.213331] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4033.760192] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4035.066215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4035.070163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4035.090444] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4035.106840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4035.163237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4036.434781] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4036.502712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4040.161329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.163607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.164148] LustreError: Skipped 1 previous similar message [ 4040.164606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.168425] Lustre: Skipped 1 previous similar message [ 4043.214324] Lustre: server umount lustre-OST0000 complete [ 4044.767938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4044.771078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.775718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.778055] Lustre: Skipped 1 previous similar message [ 4050.400178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.403057] Lustre: Skipped 3 previous similar messages [ 4050.523137] Lustre: server umount lustre-MDT0000 complete [ 4051.706035] LustreError: 285094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791694 with bad export cookie 17405701790050512813 [ 4051.709542] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.709585] LustreError: 285094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4051.819548] Lustre: server umount lustre-MDT0001 complete [ 4054.502257] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:14:57 (1785791697) [ 4056.546307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.702915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.997169] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4060.748172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.857805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.166224] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4064.504815] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.292177] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4068.471263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4070.392645] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4072.659899] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4086.239755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.242566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.246543] Lustre: Skipped 1 previous similar message [ 4086.247910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4091.985142] Lustre: server umount lustre-OST0000 complete [ 4096.992146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4096.993100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.995844] LustreError: Skipped 2 previous similar messages [ 4097.000166] Lustre: Skipped 1 previous similar message [ 4097.000735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.006292] Lustre: Skipped 2 previous similar messages [ 4099.414638] Lustre: server umount lustre-MDT0000 complete [ 4100.676584] LustreError: 289414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791743 with bad export cookie 17405701790050513681 [ 4100.680312] LustreError: 290170:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4100.681094] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4100.683641] LustreError: 290170:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2361b25 x1872538256861568/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4100.811828] Lustre: server umount lustre-MDT0001 complete [ 4104.446187] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4105.391103] Key type lgssc unregistered [ 4105.516370] LNet: 292984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4105.518243] LNetError: 292984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4105.523301] LNet: Removed LNI 192.168.206.121@tcp [ 4105.807551] Key type .llcrypt unregistered [ 4105.809136] Key type ._llcrypt unregistered [ 4113.881526] Key type ._llcrypt registered [ 4113.882662] Key type .llcrypt registered [ 4113.917105] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4119.816267] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4120.144233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4120.183557] alg: No test for adler32 (adler32-zlib) [ 4121.032212] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4121.113041] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4122.687177] Key type lgssc registered [ 4123.023933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4137.153887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4137.162189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.254764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4138.263989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4138.295792] Lustre: lustre-MDT0000: new disk, initializing [ 4138.315290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.320587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4139.615583] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4143.596105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.634875] Lustre: 296781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4143.658895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4143.661210] Lustre: Skipped 1 previous similar message [ 4143.693538] Lustre: lustre-MDT0001: new disk, initializing [ 4143.721890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.733840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4143.737271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4144.941127] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4148.575568] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4150.417165] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.623965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.725383] Lustre: lustre-OST2710: new disk, initializing [ 4152.726868] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4152.729224] Lustre: 298745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4152.760672] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4154.703683] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4158.252534] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4158.965056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4158.968340] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4158.999678] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4160.358762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4160.416757] Lustre: lustre-OST03e8: new disk, initializing [ 4160.418183] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4160.420143] Lustre: 299811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4160.449159] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4162.158588] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4164.278636] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4166.726466] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4166.797831] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4168.049785] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4168.126594] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4168.715116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4168.719413] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4168.755859] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4169.478431] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4169.548868] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4170.908338] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4170.979612] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4173.792803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.794768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4173.796910] Lustre: Skipped 3 previous similar messages [ 4173.798958] Lustre: Skipped 2 previous similar messages [ 4178.008827] Lustre: server umount lustre-MDT0000 complete [ 4179.199130] LustreError: 299810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791822 with bad export cookie 4234941671600518549 [ 4179.202583] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.203292] LustreError: 299810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4179.314169] Lustre: server umount lustre-MDT0001 complete [ 4190.211878] Lustre: server umount lustre-OST2710 complete [ 4201.622789] Lustre: server umount lustre-OST03e8 complete [ 4207.083269] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4209.838640] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4220.795657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.869537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4220.878224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4220.910428] Lustre: lustre-MDT0000: new disk, initializing [ 4220.930530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.935535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4222.156318] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4226.020862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.057258] Lustre: 305046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4226.149698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4227.355196] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4230.860681] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4232.465829] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.532662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.615221] Lustre: lustre-OST0000: new disk, initializing [ 4234.616319] Lustre: Skipped 1 previous similar message [ 4234.617512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4234.618871] Lustre: Skipped 2 previous similar messages [ 4234.620753] Lustre: 307014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4234.646986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.650046] Lustre: Skipped 1 previous similar message [ 4236.321817] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4239.768559] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4240.882559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4240.884673] Lustre: Skipped 1 previous similar message [ 4240.885994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4240.920040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4241.079985] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4241.139918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.347096] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.408219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4245.985913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4245.986266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.989121] LustreError: Skipped 1 previous similar message [ 4245.992981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.994990] Lustre: Skipped 2 previous similar messages [ 4249.100472] Lustre: server umount lustre-OST0000 complete [ 4251.104789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.105321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.108415] Lustre: Skipped 2 previous similar messages [ 4256.224028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4256.225835] Lustre: Skipped 2 previous similar messages [ 4256.405602] Lustre: server umount lustre-MDT0000 complete [ 4257.570062] LustreError: 305038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791900 with bad export cookie 4234941671600520222 [ 4257.572889] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.573760] LustreError: 305038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4257.686518] Lustre: server umount lustre-MDT0001 complete [ 4260.187832] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:18:23 (1785791903) [ 4260.678534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4261.228689] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:18:24 (1785791904) [ 4262.242440] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4270.563096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.639516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4270.646612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4270.696784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.899401] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4274.497453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.522613] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4274.532649] Lustre: 310865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4275.785279] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4291.362068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4291.427623] Lustre: Denying initial registration attempt from nid 192.168.206.121@tcp, specified as failover [ 4291.429949] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4291.432804] LustreError: 312310:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4291.435536] LustreError: 312310:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4291.438731] LustreError: 312310:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4291.459903] Lustre: server umount lustre-OST0000 complete [ 4291.460969] LustreError: 312310:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4294.624743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.625685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4294.627773] Lustre: Skipped 1 previous similar message [ 4299.600118] Lustre: server umount lustre-MDT0000 complete [ 4299.744788] LustreError: 310874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.748763] LustreError: 310874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4300.723962] LustreError: 310857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791943 with bad export cookie 4234941671600521076 [ 4300.726835] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.727599] LustreError: 310857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4300.830432] Lustre: server umount lustre-MDT0001 complete [ 4304.028703] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4304.972114] Key type lgssc unregistered [ 4305.096371] LNet: 313436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4305.098814] LNetError: 313436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4305.109314] LNet: Removed LNI 192.168.206.121@tcp [ 4305.383423] Key type .llcrypt unregistered [ 4305.385101] Key type ._llcrypt unregistered [ 4308.554822] Key type ._llcrypt registered [ 4308.555736] Key type .llcrypt registered [ 4308.589191] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4308.993859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4309.000177] alg: No test for adler32 (adler32-zlib) [ 4309.848392] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4309.929334] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4311.503127] Key type lgssc registered [ 4311.831369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4335.550144] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4338.633280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4338.640870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.744449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4339.754143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4339.816580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.280146] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4344.009461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.039874] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4344.048983] Lustre: 316175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4344.133864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4345.432569] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4347.736310] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4349.459240] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4364.642036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.709431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4364.775560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.715265] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4369.205648] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4369.982736] Lustre: server umount lustre-OST0000 complete [ 4371.952057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.953170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4371.955843] Lustre: Skipped 1 previous similar message [ 4371.960071] Lustre: Skipped 1 previous similar message [ 4377.056374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.058383] Lustre: Skipped 1 previous similar message [ 4377.428498] Lustre: server umount lustre-MDT0000 complete [ 4378.632173] LustreError: 316166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792021 with bad export cookie 18110238405059454077 [ 4378.635363] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.636233] LustreError: 316166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4378.747231] Lustre: server umount lustre-MDT0001 complete [ 4382.246426] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4383.303991] Key type lgssc unregistered [ 4383.436400] LNet: 319734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4383.439944] LNetError: 319734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4383.447387] LNet: Removed LNI 192.168.206.121@tcp [ 4383.763076] Key type .llcrypt unregistered [ 4383.764137] Key type ._llcrypt unregistered [ 4393.104715] Key type ._llcrypt registered [ 4393.106158] Key type .llcrypt registered [ 4393.151388] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4393.517612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.564562] alg: No test for adler32 (adler32-zlib) [ 4394.406335] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4394.483914] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4396.063135] Key type lgssc registered [ 4396.409149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.822862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.827790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.965871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.191978] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4403.778162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.892263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.126742] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4407.292494] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4408.902454] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4413.090409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4413.213073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.138859] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4417.557942] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4430.817500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.818475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4430.824226] Lustre: Skipped 2 previous similar messages [ 4430.825856] Lustre: Skipped 1 previous similar message [ 4433.394061] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 4433.396071] Lustre: Skipped 1 previous similar message [ 4435.936575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.938702] Lustre: Skipped 2 previous similar messages [ 4436.558854] Lustre: server umount lustre-MDT0000 complete [ 4437.755190] LustreError: 321113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792080 with bad export cookie 4359020323948543400 [ 4437.758371] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.758571] LustreError: 321113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4438.506718] LustreError: 321129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.513414] LustreError: 321129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4438.514596] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 4441.055675] LustreError: 321128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.055900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.079828] LustreError: 322658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.627848] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 4443.630901] Lustre: Skipped 2 previous similar messages [ 4447.199733] LustreError: 321129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.204791] LustreError: 321129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4452.319095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4452.319807] LustreError: 321127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.319976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4452.319979] Lustre: Skipped 2 previous similar messages [ 4452.331384] LustreError: 321127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4452.400158] Lustre: server umount lustre-MDT0001 complete [ 4454.518701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4457.321787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.559488] LustreError: 320444:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002c59953c x1872538671015424/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4462.658970] LustreError: 322959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.679179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.697264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4463.958680] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4466.617640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.737603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4468.080898] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4470.409045] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.028597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4480.614571] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4481.073709] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4482.022070] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4483.039894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4483.042610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.047935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.141404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.143481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.404578] Lustre: server umount lustre-OST0000 complete [ 4492.256518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.256596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4495.696895] Lustre: server umount lustre-MDT0000 complete [ 4496.934270] LustreError: 321113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792139 with bad export cookie 4359020323948550519 [ 4496.938397] LustreError: 321113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4496.938531] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4497.063219] Lustre: server umount lustre-MDT0001 complete [ 4500.588169] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4501.593120] Key type lgssc unregistered [ 4501.733336] LNet: 327412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4501.735775] LNetError: 327412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4501.745278] LNet: Removed LNI 192.168.206.121@tcp [ 4502.043365] Key type .llcrypt unregistered [ 4502.044262] Key type ._llcrypt unregistered [ 4510.686793] Key type ._llcrypt registered [ 4510.688050] Key type .llcrypt registered [ 4510.722461] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4511.110644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4511.121078] alg: No test for adler32 (adler32-zlib) [ 4511.968136] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4512.046700] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4513.623127] Key type lgssc registered [ 4513.951736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4516.359779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4516.366276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.502177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.814558] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4521.543141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4521.662352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.020782] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4525.252165] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4526.878041] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4531.873252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.880080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4531.881894] Lustre: Skipped 1 previous similar message [ 4533.909084] Lustre: server umount lustre-MDT0000 complete [ 4535.063193] LustreError: 328785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792178 with bad export cookie 1364389176275392662 [ 4535.066381] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.067384] LustreError: 328785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.172599] Lustre: server umount lustre-MDT0001 complete [ 4539.743450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.831678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4539.845035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4539.911893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.101539] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4543.630343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.661174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4543.671570] Lustre: 331389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4543.754373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4544.954038] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4547.173464] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.826948] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4553.250182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4553.320616] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4553.323876] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4553.327422] LustreError: 333270:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4553.330218] LustreError: 333270:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4553.332431] LustreError: 333270:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4553.357246] Lustre: server umount lustre-OST0000 complete [ 4553.358805] LustreError: 333270:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4557.985759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4558.055896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4558.122048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.055744] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4562.512954] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4563.986486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4565.016953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.049545] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4565.087980] Lustre: lustre-OST0001: new disk, initializing [ 4565.089842] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4565.092316] Lustre: 334811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4566.894101] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4569.099624] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4570.035819] Lustre: server umount lustre-OST0001 complete [ 4573.163961] LustreError: 333738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.166950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.168729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.168797] LustreError: 333738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4573.171701] Lustre: Skipped 2 previous similar messages [ 4577.289134] Lustre: server umount lustre-OST0000 complete [ 4578.546195] Lustre: server umount lustre-MDT0000 complete [ 4579.728181] LustreError: 331381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792222 with bad export cookie 1364389176275393201 [ 4579.731990] LustreError: 331381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4579.732260] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4585.902618] Lustre: server umount lustre-MDT0001 complete [ 4589.496500] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4590.463202] Key type lgssc unregistered [ 4590.581314] LNet: 336764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4590.583567] LNetError: 336764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4590.595234] LNet: Removed LNI 192.168.206.121@tcp [ 4590.865997] Key type .llcrypt unregistered [ 4590.866870] Key type ._llcrypt unregistered [ 4620.376916] Key type ._llcrypt registered [ 4620.377921] Key type .llcrypt registered [ 4620.409782] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4626.183094] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4626.726435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4626.738053] alg: No test for adler32 (adler32-zlib) [ 4627.603330] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4627.696159] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4629.279144] Key type lgssc registered [ 4629.645833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4640.518696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4640.524893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.607484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4641.618880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4641.659151] Lustre: lustre-MDT0000: new disk, initializing [ 4641.680333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.686788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4642.961867] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4646.893859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.938866] Lustre: 341663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4646.960486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4646.962547] Lustre: Skipped 1 previous similar message [ 4646.995590] Lustre: lustre-MDT0001: new disk, initializing [ 4647.025299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.036078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4647.039885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4648.237420] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4651.861357] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4653.655175] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4655.946072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4656.046908] Lustre: lustre-OST0000: new disk, initializing [ 4656.048519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4656.051312] Lustre: 343630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4656.079910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.018827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4657.023800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4657.063447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4658.109728] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4661.872299] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.218759] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4663.290340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4664.582623] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4664.648616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4667.363334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.363939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4667.364728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.367109] Lustre: Skipped 1 previous similar message [ 4667.368937] LustreError: Skipped 1 previous similar message [ 4671.375418] Lustre: server umount lustre-OST0000 complete [ 4672.991846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4672.994870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.000202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4673.002777] Lustre: Skipped 1 previous similar message [ 4677.600261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.600876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.753491] Lustre: server umount lustre-MDT0000 complete [ 4679.977139] LustreError: 341657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792323 with bad export cookie 660853480311310224 [ 4679.979796] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4679.982013] LustreError: 341657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4680.096541] Lustre: server umount lustre-MDT0001 complete [ 4682.829041] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:25:26 (1785792326) [ 4685.083026] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:25:28 (1785792328) [ 4687.649794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.818673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.087905] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4691.849194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.988993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4693.334756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4695.673872] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.407766] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4699.546446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4701.458376] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4703.720739] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4717.790177] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4718.347025] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4718.889836] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4719.438613] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4720.039156] Lustre: Failing over lustre-MDT0000 [ 4720.250664] Lustre: server umount lustre-MDT0000 complete [ 4722.655853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4722.659037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.664151] LustreError: 346392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.670525] LustreError: 346392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4723.168325] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.173691] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4724.722077] LustreError: 348754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.290344] LustreError: 346388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.294933] LustreError: 346388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4732.923868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.962964] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.045842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.047896] Lustre: Skipped 1 previous similar message [ 4733.069799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4734.225503] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4734.955211] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4738.532863] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4738.541819] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4738.558363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4740.151149] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4740.636789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4742.384833] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4742.958711] Lustre: Failing over lustre-MDT0000 [ 4743.183735] Lustre: server umount lustre-MDT0000 complete [ 4743.648037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4743.649668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.650306] LustreError: 347125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.650311] LustreError: 347125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4743.665015] Lustre: Skipped 4 previous similar messages [ 4753.889288] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.894986] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4755.916273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.957191] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4756.045913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.068244] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4757.398060] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4760.556075] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4761.064289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4761.066476] Lustre: Skipped 2 previous similar messages [ 4761.076600] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4761.095862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4762.954750] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4763.483265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4766.177970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.178064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4766.178799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4766.178804] Lustre: Skipped 1 previous similar message [ 4766.181702] Lustre: Skipped 1 previous similar message [ 4771.296495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4771.299258] Lustre: Skipped 4 previous similar messages [ 4771.862920] Lustre: server umount lustre-MDT0000 complete [ 4773.031988] LustreError: 350374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792416 with bad export cookie 660853480311312905 [ 4773.035269] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4773.036475] LustreError: 350374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4773.143633] Lustre: server umount lustre-MDT0001 complete [ 4784.538967] Lustre: server umount lustre-OST0000 complete [ 4789.365048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.501280] LustreError: 352741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.506376] LustreError: 352741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4789.522540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.828197] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4793.536266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.004182] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4797.262373] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4798.914144] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4801.020873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4802.950874] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4805.282543] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4815.462342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4818.911664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4818.915223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.920722] Lustre: Skipped 1 previous similar message [ 4818.923092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4823.116920] Lustre: server umount lustre-OST0000 complete [ 4830.487580] Lustre: server umount lustre-MDT0000 complete [ 4831.200075] LustreError: 352741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.209899] LustreError: 352741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4831.749126] LustreError: 352723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792474 with bad export cookie 660853480311313626 [ 4831.752526] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4831.754262] LustreError: 352723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4831.873188] Lustre: server umount lustre-MDT0001 complete [ 4835.480093] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4836.424484] Key type lgssc unregistered [ 4836.538386] LNet: 356290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4836.541091] LNetError: 356290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4836.547259] LNet: Removed LNI 192.168.206.121@tcp [ 4836.821602] Key type .llcrypt unregistered [ 4836.823202] Key type ._llcrypt unregistered [ 4846.343347] Key type ._llcrypt registered [ 4846.345052] Key type .llcrypt registered [ 4846.385864] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4846.716818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4846.772250] alg: No test for adler32 (adler32-zlib) [ 4847.623483] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4847.716453] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4849.311118] Key type lgssc registered [ 4849.694155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4852.482815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4852.489486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.649370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.111899] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4858.055678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.227427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.610353] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4862.063718] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4863.898805] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4866.282582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.430302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.387969] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4870.797425] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4878.758057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4889.056839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.058989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4889.059409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.060918] Lustre: Skipped 2 previous similar messages [ 4891.220839] Lustre: server umount lustre-MDT0000 complete [ 4892.424124] LustreError: 357765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792535 with bad export cookie 16194301962027709315 [ 4892.427596] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4892.428178] LustreError: 357765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4892.539501] Lustre: server umount lustre-MDT0001 complete [ 4903.954127] Lustre: server umount lustre-OST0000 complete [ 4908.141743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.275645] LustreError: 361513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.280119] LustreError: 361513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4908.296347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.614269] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4912.248308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.349249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4913.589098] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4915.875317] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4917.586663] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4919.930380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.933049] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4924.303747] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4933.349198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4943.327845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4943.330599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.335609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.338555] Lustre: Skipped 2 previous similar messages [ 4947.543965] Lustre: server umount lustre-MDT0000 complete [ 4948.819234] LustreError: 364126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792591 with bad export cookie 16194301962027710414 [ 4948.823576] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.824830] LustreError: 364126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4948.945342] Lustre: server umount lustre-MDT0001 complete [ 4960.473995] Lustre: server umount lustre-OST0000 complete [ 4965.666626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.825578] LustreError: 365439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.849569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4965.852759] Lustre: Skipped 1 previous similar message [ 4967.243421] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4970.214365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.721178] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4974.164543] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4976.084889] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4978.383222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.514307] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4983.021783] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4990.821940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4996.064626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4996.066728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.068309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.071970] Lustre: Skipped 3 previous similar messages [ 4996.073940] Lustre: Skipped 2 previous similar messages [ 5000.719629] Lustre: server umount lustre-OST0000 complete [ 5006.305546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.306087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5006.306568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5006.306573] Lustre: Skipped 1 previous similar message [ 5006.309198] Lustre: Skipped 1 previous similar message [ 5006.318613] LustreError: Skipped 1 previous similar message [ 5008.016495] Lustre: server umount lustre-MDT0000 complete [ 5009.155069] LustreError: 365421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792652 with bad export cookie 16194301962027711422 [ 5009.157819] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.159099] LustreError: 365421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5009.267970] Lustre: server umount lustre-MDT0001 complete [ 5012.510526] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5013.502895] Key type lgssc unregistered [ 5013.636423] LNet: 368988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5013.639372] LNetError: 368988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5013.649294] LNet: Removed LNI 192.168.206.121@tcp [ 5013.947093] Key type .llcrypt unregistered [ 5013.948069] Key type ._llcrypt unregistered [ 5022.740316] Key type ._llcrypt registered [ 5022.741449] Key type .llcrypt registered [ 5022.779797] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5023.186268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5023.241768] alg: No test for adler32 (adler32-zlib) [ 5024.100448] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5024.182323] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5025.767155] Key type lgssc registered [ 5026.134287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5028.688372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5028.696478] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.700764] LustreError: 370405:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5028.710045] LustreError: 370405:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5028.712763] LustreError: 370405:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5028.715446] LustreError: 370405:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5028.718394] LustreError: 370405:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5031.143412] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.145521] LustreError: 370802:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5031.152743] LustreError: 370802:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5031.155302] LustreError: 370802:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5031.158895] LustreError: 370802:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5031.161084] LustreError: 370802:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5036.967476] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5037.890903] Key type lgssc unregistered [ 5038.011310] LNet: 371802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5038.014193] LNetError: 371802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5038.023321] LNet: Removed LNI 192.168.206.121@tcp [ 5038.298278] Key type .llcrypt unregistered [ 5038.299538] Key type ._llcrypt unregistered [ 5044.158101] Key type ._llcrypt registered [ 5044.159512] Key type .llcrypt registered [ 5044.196099] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 5050.237630] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5050.649940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5050.675217] alg: No test for adler32 (adler32-zlib) [ 5051.543720] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5051.635738] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5053.215158] Key type lgssc registered [ 5053.584257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.501188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5064.509870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.588161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5065.599362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5065.636416] Lustre: lustre-MDT0000: new disk, initializing [ 5065.656164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.662302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5066.964126] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5070.973480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.017481] Lustre: 374930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.043601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5071.045498] Lustre: Skipped 1 previous similar message [ 5071.077677] Lustre: lustre-MDT0001: new disk, initializing [ 5071.108061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.120815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5071.123489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5072.380919] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5075.898793] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5077.614702] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5079.802038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5079.898206] Lustre: lustre-OST0000: new disk, initializing [ 5079.899747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5079.902124] Lustre: 376895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5079.928918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.669398] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5082.292480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5082.295689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5082.323579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5085.188927] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5086.566323] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5086.630084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5087.939141] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5088.007363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5092.835302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.837023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.837481] LustreError: Skipped 1 previous similar message [ 5092.843190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.846230] Lustre: Skipped 1 previous similar message [ 5094.733827] Lustre: server umount lustre-OST0000 complete [ 5096.927721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.930484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.934859] Lustre: Skipped 1 previous similar message [ 5096.937194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.952153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.952724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.071952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.073351] Lustre: Skipped 2 previous similar messages [ 5108.192274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.194552] Lustre: Skipped 1 previous similar message [ 5110.239150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5110.314388] Lustre: server umount lustre-MDT0000 complete [ 5111.596295] LustreError: 374922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792754 with bad export cookie 7673804226654849669 [ 5111.599715] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.601937] LustreError: 374922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5111.730751] Lustre: server umount lustre-MDT0001 complete [ 5114.461827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5114.980941] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:32:38 (1785792758) [ 5117.033327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.188060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.543098] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5121.428289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.560959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.860614] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5125.193313] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5126.899052] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5129.072328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5130.963326] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5133.214291] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5135.390542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.443697] Lustre: lustre-OST0001: new disk, initializing [ 5135.445085] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5135.447609] Lustre: 382186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5135.472540] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.475231] Lustre: Skipped 1 previous similar message [ 5137.261852] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5137.403905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5137.407757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5137.432918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5140.928951] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5147.121086] Lustre: lustre-OST0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 5147.617968] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.618537] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.620439] LustreError: Skipped 1 previous similar message [ 5148.796896] Lustre: server umount lustre-OST0001 complete [ 5152.739787] LustreError: 381174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.746417] LustreError: 381174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5157.859625] LustreError: 381172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.866176] LustreError: 381172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5161.951773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5161.955266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.958728] Lustre: Skipped 1 previous similar message [ 5162.978613] LustreError: 381173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.985227] LustreError: 381173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5168.097048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5168.097749] LustreError: 381172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.099288] Lustre: Skipped 6 previous similar messages [ 5168.103467] LustreError: 381172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5173.219448] LustreError: 381174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5173.225819] LustreError: 381174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5175.263103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5175.317630] Lustre: server umount lustre-OST0000 complete [ 5177.824180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5177.826663] LustreError: Skipped 1 previous similar message [ 5177.828715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.833827] Lustre: Skipped 1 previous similar message [ 5182.949112] Lustre: server umount lustre-MDT0000 complete [ 5183.456096] LustreError: 379352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.460468] LustreError: 379352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5184.164372] LustreError: 379380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792827 with bad export cookie 7673804226654850537 [ 5184.169661] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5190.333219] Lustre: server umount lustre-MDT0001 complete [ 5193.755040] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5194.710179] Key type lgssc unregistered [ 5194.833443] LNet: 384171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5194.835657] LNetError: 384171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5194.845285] LNet: Removed LNI 192.168.206.121@tcp [ 5195.108856] Key type .llcrypt unregistered [ 5195.109893] Key type ._llcrypt unregistered [ 5212.445734] Key type ._llcrypt registered [ 5212.447444] Key type .llcrypt registered [ 5212.482880] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5212.906833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5212.921929] alg: No test for adler32 (adler32-zlib) [ 5213.774568] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5213.857544] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5215.431147] Key type lgssc registered [ 5215.780050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5218.255786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5218.261843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.342276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5219.349112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5219.399792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.704172] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5223.520330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.559063] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5223.569141] Lustre: 386946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5223.675427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.999200] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5227.272717] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5229.044785] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5233.632609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.633177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.636373] Lustre: Skipped 1 previous similar message [ 5235.856700] Lustre: server umount lustre-MDT0000 complete [ 5236.973114] LustreError: 386938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792880 with bad export cookie 1121405564130784263 [ 5236.975695] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5236.975787] LustreError: 386938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5237.074692] Lustre: server umount lustre-MDT0001 complete [ 5239.858683] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:34:43 (1785792883) [ 5241.879960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.030955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.252769] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5245.911549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.027925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.352161] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5249.660277] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5251.412203] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5253.504688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.566896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5253.624967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.407247] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5257.655448] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5266.727524] Lustre: Permanently deactivating lustre-OST0000 [ 5266.732573] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5268.204519] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5268.536361] LustreError: 392294:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5268.539217] LustreError: 392294:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5268.541273] LustreError: 392294:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5269.079809] LustreError: 392342:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5269.082065] LustreError: 392342:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5275.790442] Lustre: server umount lustre-OST0000 complete [ 5276.627700] LustreError: 392592:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5276.629635] LustreError: 392592:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5279.712508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.713022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.716739] Lustre: Skipped 1 previous similar message [ 5279.719574] Lustre: Skipped 2 previous similar messages [ 5284.832647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5284.835232] Lustre: Skipped 1 previous similar message [ 5287.930688] Lustre: server umount lustre-MDT0000 complete [ 5289.098194] LustreError: 391288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792932 with bad export cookie 1121405564130784830 [ 5289.101788] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.104230] LustreError: 391288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5299.731654] Lustre: server umount lustre-MDT0001 complete [ 5302.657065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.993250] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5304.905838] LustreError: 393749:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5307.025895] Lustre: 394087:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.121@tcp [ 5307.717053] Lustre: server umount MGS complete [ 5311.505432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.683710] LustreError: 394810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.688685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5311.691414] LustreError: 394810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5311.693853] Lustre: Skipped 1 previous similar message [ 5311.715484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.073821] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5315.817468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.914545] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5317.266693] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5319.608397] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5321.307368] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5321.901300] Lustre: Permanently reactivating lustre-OST0000 [ 5321.905775] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5321.908512] Lustre: Skipped 2 previous similar messages [ 5343.711154] LustreError: 396451:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5346.084807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.213554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5346.216154] Lustre: Skipped 1 previous similar message [ 5348.099419] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5350.402107] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5363.681853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5363.681859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.682600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.683731] LustreError: Skipped 1 previous similar message [ 5364.362665] Lustre: server umount lustre-OST0000 complete [ 5367.263873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.265875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.269432] Lustre: Skipped 1 previous similar message [ 5367.575422] Lustre: server umount lustre-MDT0000 complete [ 5368.751928] LustreError: 396383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793011 with bad export cookie 1121405564130785852 [ 5368.754864] LustreError: 396383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5368.755283] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.863345] Lustre: server umount lustre-MDT0001 complete [ 5372.325883] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5373.288143] Key type lgssc unregistered [ 5373.415286] LNet: 399334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5373.417307] LNetError: 399334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5373.425282] LNet: Removed LNI 192.168.206.121@tcp [ 5373.713959] Key type .llcrypt unregistered [ 5373.715626] Key type ._llcrypt unregistered [ 5379.498830] Key type ._llcrypt registered [ 5379.500429] Key type .llcrypt registered [ 5379.537822] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 5385.322158] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5385.662718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5385.690946] alg: No test for adler32 (adler32-zlib) [ 5386.538257] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5386.621069] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5388.199152] Key type lgssc registered [ 5388.517502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5399.307375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5399.312457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.395933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5400.405952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5400.443073] Lustre: lustre-MDT0000: new disk, initializing [ 5400.465166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.472047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5401.832359] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5405.918493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.959761] Lustre: 402463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5405.986250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5405.988302] Lustre: Skipped 1 previous similar message [ 5406.021305] Lustre: lustre-MDT0001: new disk, initializing [ 5406.059471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.073629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5406.077949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5407.471124] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5410.977108] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5412.548872] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5414.579826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5414.684286] Lustre: lustre-OST0000: new disk, initializing [ 5414.685808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5414.688636] Lustre: 404428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5414.713674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.351268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5415.353773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5415.381320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5416.464124] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5419.982287] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5421.325198] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5421.386743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5422.739788] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5422.812173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5425.633444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5425.635631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.637531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5425.640762] Lustre: Skipped 1 previous similar message [ 5429.517269] Lustre: server umount lustre-OST0000 complete [ 5430.752243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.752763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.756648] Lustre: Skipped 1 previous similar message [ 5430.759472] Lustre: Skipped 2 previous similar messages [ 5435.872056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5435.873785] Lustre: Skipped 1 previous similar message [ 5440.992063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5440.994343] Lustre: Skipped 1 previous similar message [ 5445.087150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5445.088575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5445.092165] Lustre: Skipped 1 previous similar message [ 5445.150877] Lustre: server umount lustre-MDT0000 complete [ 5446.347177] LustreError: 403256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793089 with bad export cookie 15299502144881648621 [ 5446.350509] LustreError: 403256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5446.350765] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5446.461308] Lustre: server umount lustre-MDT0001 complete [ 5449.160526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5449.674632] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:38:13 (1785793093) [ 5451.818503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.011406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.315642] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5456.188871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.343849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5457.717533] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5460.096891] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.937406] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5464.303762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.455517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.738343] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5469.372272] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.018304] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5481.004172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5482.977765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5482.977949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.979910] LustreError: Skipped 1 previous similar message [ 5482.984841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5482.987238] Lustre: Skipped 1 previous similar message [ 5488.467249] Lustre: server umount lustre-OST0000 complete [ 5492.191760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5492.193459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.196256] Lustre: Skipped 1 previous similar message [ 5495.765509] Lustre: server umount lustre-MDT0000 complete [ 5496.982350] LustreError: 409587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793140 with bad export cookie 15299502144881649489 [ 5496.986459] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.125481] Lustre: server umount lustre-MDT0001 complete [ 5500.515967] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5501.447203] Key type lgssc unregistered [ 5501.569422] LNet: 410628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5501.572401] LNetError: 410628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5501.583426] LNet: Removed LNI 192.168.206.121@tcp [ 5501.924385] Key type .llcrypt unregistered [ 5501.925278] Key type ._llcrypt unregistered [ 5509.086170] Key type ._llcrypt registered [ 5509.087340] Key type .llcrypt registered [ 5509.122490] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5509.212432] Key type .llcrypt unregistered [ 5509.213760] Key type ._llcrypt unregistered [ 5515.833517] Key type ._llcrypt registered [ 5515.834611] Key type .llcrypt registered [ 5515.871744] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5516.269450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5516.276828] alg: No test for adler32 (adler32-zlib) [ 5517.117961] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5517.195251] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5518.767161] Key type lgssc registered [ 5519.145392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5521.757211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5521.762597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.918817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.340644] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5527.233712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.361249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.247159] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5531.611649] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5532.512967] LustreError: 412817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5532.517933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5532.520613] LustreError: 412817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5533.930123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.059605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.485284] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5542.879973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.882707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.886992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.420239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.420828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.421091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5548.557624] Lustre: server umount lustre-OST0000 complete [ 5553.121421] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5553.123604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5553.127256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5553.128867] Lustre: Skipped 1 previous similar message [ 5555.997090] Lustre: server umount lustre-MDT0000 complete [ 5557.286340] LustreError: 414926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793200 with bad export cookie 13056427673047645910 [ 5557.291128] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5557.291170] LustreError: 414926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5557.404716] Lustre: server umount lustre-MDT0001 complete [ 5561.061776] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5562.089209] Key type lgssc unregistered [ 5562.226455] LNet: 415965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5562.230406] LNetError: 415965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5562.239292] LNet: Removed LNI 192.168.206.121@tcp [ 5562.547188] Key type .llcrypt unregistered [ 5562.548679] Key type ._llcrypt unregistered [ 5571.584976] Key type ._llcrypt registered [ 5571.585952] Key type .llcrypt registered [ 5571.622737] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5571.921131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5571.971026] alg: No test for adler32 (adler32-zlib) [ 5572.815196] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5572.896974] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5574.479159] Key type lgssc registered [ 5574.826343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5577.420628] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5577.429920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.863278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5597.983276] LustreError: 417331:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.121@tcp: failed processing log, type 4: rc = -110 [ 5624.607287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.544492] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5628.930238] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5631.147424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.312758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.339440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5632.657415] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5635.555319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.693878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5637.109096] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5639.541649] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5641.419932] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.282524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.283313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5645.286747] Lustre: Skipped 1 previous similar message [ 5650.400940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.403016] Lustre: Skipped 2 previous similar messages [ 5650.700101] Lustre: server umount lustre-OST0000 complete [ 5655.520470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.525571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.527864] Lustre: Skipped 1 previous similar message [ 5657.938210] Lustre: server umount lustre-MDT0000 complete [ 5659.108163] LustreError: 419108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793302 with bad export cookie 12746828705825998758 [ 5659.111703] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.112342] LustreError: 419108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5659.226474] Lustre: server umount lustre-MDT0001 complete [ 5662.499931] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5663.550312] Key type lgssc unregistered [ 5663.676344] LNet: 420901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.679029] LNetError: 420901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.685253] LNet: Removed LNI 192.168.206.121@tcp [ 5663.998594] Key type .llcrypt unregistered [ 5663.999949] Key type ._llcrypt unregistered [ 5672.744367] Key type ._llcrypt registered [ 5672.745421] Key type .llcrypt registered [ 5672.781104] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5673.070774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.095053] alg: No test for adler32 (adler32-zlib) [ 5673.929273] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5674.003407] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5675.575146] Key type lgssc registered [ 5675.972083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5678.649569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.655422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.798390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.211870] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5683.873433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.987541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5685.288876] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5687.470197] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5689.100517] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5691.179176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.286619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.088484] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5695.445526] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5704.615041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5709.796817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.797649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5709.797924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.800990] Lustre: Skipped 1 previous similar message [ 5710.836033] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 5710.838024] Lustre: Skipped 2 previous similar messages [ 5712.788506] Lustre: server umount lustre-MDT0000 complete [ 5713.471597] LustreError: 422274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.21@tcp arrived at 1785793356 with bad export cookie 11568133093646850361 [ 5713.477134] LustreError: 422274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5714.123884] LustreError: 423027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793357 with bad export cookie 11568133093646849738 [ 5714.127534] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5714.128019] LustreError: 423027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5714.912147] LustreError: 422292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.912246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.917719] LustreError: 422292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5714.919714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.924421] Lustre: Skipped 1 previous similar message [ 5714.925980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.031888] LustreError: 422293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.032171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.036095] LustreError: 422293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5720.270678] Lustre: server umount lustre-OST0000 complete [ 5722.545615] Lustre: server umount lustre-MDT0001 complete [ 5725.823436] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5726.824099] Key type lgssc unregistered [ 5726.953474] LNet: 425994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5726.956251] LNetError: 425994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5726.965502] LNet: Removed LNI 192.168.206.121@tcp [ 5727.268439] Key type .llcrypt unregistered [ 5727.269300] Key type ._llcrypt unregistered [ 5735.915242] Key type ._llcrypt registered [ 5735.916538] Key type .llcrypt registered [ 5735.959203] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5736.303749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5736.339898] alg: No test for adler32 (adler32-zlib) [ 5737.196696] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5737.291238] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5738.871175] Key type lgssc registered [ 5739.276270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5741.996800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5742.005269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.139359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.402658] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5747.120958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.236576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5748.518706] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5750.794939] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5752.518615] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5754.530847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5754.651706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.579688] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5759.057398] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5767.974062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5772.767777] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5772.770113] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.774624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5773.281609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5773.281791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.835888] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 5774.837967] Lustre: Skipped 1 previous similar message [ 5776.654343] Lustre: server umount lustre-MDT0001 complete [ 5778.400060] LustreError: 429874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.400435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.401031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.401533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5778.407125] LustreError: 429874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5783.520575] LustreError: 427386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.521827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.328985] Lustre: server umount lustre-OST0000 complete [ 5785.608408] Lustre: server umount lustre-MDT0000 complete [ 5789.875567] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5790.857365] Key type lgssc unregistered [ 5790.982382] LNet: 431091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5790.984928] LNetError: 431091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5790.994271] LNet: Removed LNI 192.168.206.121@tcp [ 5791.284981] Key type .llcrypt unregistered [ 5791.286045] Key type ._llcrypt unregistered [ 5799.045724] Key type ._llcrypt registered [ 5799.046576] Key type .llcrypt registered [ 5799.081864] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 5804.960406] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5805.270938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5805.289067] alg: No test for adler32 (adler32-zlib) [ 5806.132048] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5806.221508] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5807.799156] Key type lgssc registered [ 5808.140584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5819.502788] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5824.246554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5824.254027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.337262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5825.350065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5825.388841] Lustre: lustre-MDT0000: new disk, initializing [ 5825.408903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.415095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5826.760055] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5832.529758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.624477] Lustre: lustre-OST0000: new disk, initializing [ 5832.626101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5832.627611] Lustre: Skipped 1 previous similar message [ 5832.629320] Lustre: 436096:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5832.650932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.469069] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5834.560329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5834.563404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5834.571600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5837.980521] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5843.205957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.248077] Lustre: 435077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5843.276510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5843.311404] Lustre: lustre-MDT0001: new disk, initializing [ 5843.349827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5843.361376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5843.364058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5844.650995] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5848.093596] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5848.163720] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5849.553254] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5851.659692] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 5855.218154] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 5856.225807] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.226311] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5856.230579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5856.234462] Lustre: Skipped 1 previous similar message [ 5859.276105] Lustre: server umount lustre-MDT0001 complete [ 5866.465492] LustreError: 435084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.465492] LustreError: 435088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.465513] LustreError: 435084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5866.470078] LustreError: 435088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5871.584764] LustreError: 435089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.584798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.593241] Lustre: Skipped 1 previous similar message [ 5871.595291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.704509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.705430] LustreError: 435088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.648382] Lustre: server umount lustre-OST0000 complete [ 5885.109883] Lustre: server umount lustre-MDT0000 complete [ 5889.745951] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5890.848792] Key type lgssc unregistered [ 5890.986509] LNet: 439856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5890.989352] LNetError: 439856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5890.996352] LNet: Removed LNI 192.168.206.121@tcp [ 5891.332651] Key type .llcrypt unregistered [ 5891.333893] Key type ._llcrypt unregistered [ 5903.750491] Key type ._llcrypt registered [ 5903.751459] Key type .llcrypt registered [ 5903.797345] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5903.920331] Key type .llcrypt unregistered [ 5903.921612] Key type ._llcrypt unregistered [ 5910.163348] Key type ._llcrypt registered [ 5910.164251] Key type .llcrypt registered [ 5910.201485] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5910.496455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5910.519523] alg: No test for adler32 (adler32-zlib) [ 5911.363068] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 5911.449595] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5913.031130] Key type lgssc registered [ 5913.362914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.657331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.662874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.739650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5922.751066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5922.787253] Lustre: lustre-MDT0000: new disk, initializing [ 5922.808271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.814235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5924.105949] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5928.077638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.118783] Lustre: 443201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5928.143261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5928.144867] Lustre: Skipped 1 previous similar message [ 5928.175778] Lustre: lustre-MDT0001: new disk, initializing [ 5928.205610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5928.216697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5928.219395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5929.469361] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5932.946131] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5934.657473] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5936.707583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.810408] Lustre: lustre-OST0000: new disk, initializing [ 5936.812815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5936.816324] Lustre: 445167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5936.843352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.789790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5937.793842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5937.842280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5938.617586] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5942.192322] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5948.385305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.386644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.389416] Lustre: Skipped 1 previous similar message [ 5948.394018] Lustre: Skipped 2 previous similar messages [ 5950.681891] Lustre: server umount lustre-MDT0000 complete [ 5951.909178] LustreError: 443998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793594 with bad export cookie 15646668958895943816 [ 5951.912750] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.913254] LustreError: 443998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5952.024367] Lustre: server umount lustre-MDT0001 complete [ 5963.551327] Lustre: server umount lustre-OST0000 complete [ 5966.979501] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:46:50 (1785793610) [ 5969.406948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.562354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.771924] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5973.448698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.568137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.805716] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5976.944968] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5978.552509] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5993.826150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.894498] Lustre: Found index 0 for lustre-OST0000, updating log [ 5993.955883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.969333] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5998.056406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 5998.523833] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6003.169488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.169507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.174884] Lustre: Skipped 1 previous similar message [ 6003.178114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.181545] Lustre: Skipped 1 previous similar message [ 6004.196097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.198171] Lustre: Skipped 1 previous similar message [ 6005.783084] Lustre: server umount lustre-OST0000 complete [ 6018.267479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.268143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6018.270623] Lustre: Skipped 1 previous similar message [ 6023.648280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.650721] Lustre: Skipped 2 previous similar messages [ 6024.401892] Lustre: server umount lustre-MDT0000 complete [ 6025.667069] LustreError: 449034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793668 with bad export cookie 15646668958895945139 [ 6025.670973] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.671715] LustreError: 449034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6025.792029] Lustre: server umount lustre-MDT0001 complete [ 6028.652812] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:47:52 (1785793672) [ 6060.959226] LNet: 1 local NIs in recovery (showing 1): 192.168.206.121@tcp [ 6608.138161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.290659] LustreError: 453593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.299864] LustreError: 453593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6608.311994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.645593] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6612.368165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.826501] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6616.230196] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6618.062981] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.379219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.485434] Lustre: Found index 0 for lustre-OST0000, updating log [ 6635.574925] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6635.626319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6637.979995] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6639.649922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.654728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.749406] Lustre: server umount lustre-OST0000 complete [ 6646.946101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.950069] Lustre: Skipped 1 previous similar message [ 6652.066203] LustreError: 453593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.066554] Lustre: server umount lustre-MDT0000 complete [ 6652.070881] LustreError: 453593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6653.088359] LustreError: 454321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.092549] LustreError: 454321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6653.302176] LustreError: 453573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794296 with bad export cookie 15646668958895946210 [ 6653.305055] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6653.305796] LustreError: 453573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6653.426511] Lustre: server umount lustre-MDT0001 complete [ 6662.047243] LNet: 1 local NIs in recovery (showing 1): 192.168.206.121@tcp [ 6670.530018] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 6673.823254] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6684.960857] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:58:48 (1785794328) [ 6690.594417] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6691.658911] Key type lgssc unregistered [ 6695.903415] LNet: 460244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6695.905301] LNetError: 460244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6764.517310] LNet: Removed LNI 192.168.206.121@tcp [ 6764.858294] Key type .llcrypt unregistered [ 6764.859590] Key type ._llcrypt unregistered [ 6771.927873] Key type ._llcrypt registered [ 6771.928986] Key type .llcrypt registered [ 6771.973499] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6772.383809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6772.408868] alg: No test for adler32 (adler32-zlib) [ 6773.285166] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6773.392211] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6774.975208] Key type lgssc registered [ 6775.358699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6776.773249] LNet: Added LNI 192.168.206.121@tcp100 [8/256/0/180] [ 6777.065899] LNet: Added LNI 192.168.206.121@tcp101 [8/256/0/180] [ 6777.355789] LNet: Added LNI 192.168.206.121@tcp102 [8/256/0/180] [ 6777.646574] LNet: Added LNI 192.168.206.121@tcp103 [8/256/0/180] [ 6777.949651] LNet: Added LNI 192.168.206.121@tcp104 [8/256/0/180] [ 6778.236160] LNet: Added LNI 192.168.206.121@tcp105 [8/256/0/180] [ 6778.520056] LNet: Added LNI 192.168.206.121@tcp106 [8/256/0/180] [ 6778.805337] LNet: Added LNI 192.168.206.121@tcp107 [8/256/0/180] [ 6779.101329] LNet: Added LNI 192.168.206.121@tcp108 [8/256/0/180] [ 6779.388693] LNet: Added LNI 192.168.206.121@tcp109 [8/256/0/180] [ 6779.678417] LNet: Added LNI 192.168.206.121@tcp110 [8/256/0/180] [ 6779.976368] LNet: Added LNI 192.168.206.121@tcp111 [8/256/0/180] [ 6780.269644] LNet: Added LNI 192.168.206.121@tcp112 [8/256/0/180] [ 6780.568933] LNet: Added LNI 192.168.206.121@tcp113 [8/256/0/180] [ 6780.872387] LNet: Added LNI 192.168.206.121@tcp114 [8/256/0/180] [ 6781.174562] LNet: Added LNI 192.168.206.121@tcp115 [8/256/0/180] [ 6781.472501] LNet: Added LNI 192.168.206.121@tcp116 [8/256/0/180] [ 6781.776311] LNet: Added LNI 192.168.206.121@tcp117 [8/256/0/180] [ 6782.087078] LNet: Added LNI 192.168.206.121@tcp118 [8/256/0/180] [ 6782.372858] LNet: Added LNI 192.168.206.121@tcp119 [8/256/0/180] [ 6782.677406] LNet: Added LNI 192.168.206.121@tcp120 [8/256/0/180] [ 6782.967503] LNet: Added LNI 192.168.206.121@tcp121 [8/256/0/180] [ 6783.264064] LNet: Added LNI 192.168.206.121@tcp122 [8/256/0/180] [ 6783.553632] LNet: Added LNI 192.168.206.121@tcp123 [8/256/0/180] [ 6783.836310] LNet: Added LNI 192.168.206.121@tcp124 [8/256/0/180] [ 6784.146782] LNet: Added LNI 192.168.206.121@tcp125 [8/256/0/180] [ 6784.451814] LNet: Added LNI 192.168.206.121@tcp126 [8/256/0/180] [ 6784.744121] LNet: Added LNI 192.168.206.121@tcp127 [8/256/0/180] [ 6785.035475] LNet: Added LNI 192.168.206.121@tcp128 [8/256/0/180] [ 6785.334419] LNet: Added LNI 192.168.206.121@tcp129 [8/256/0/180] [ 6785.628605] LNet: Added LNI 192.168.206.121@tcp130 [8/256/0/180] [ 6785.918962] LNet: Added LNI 192.168.206.121@tcp131 [8/256/0/180] [ 6786.209719] LNet: Added LNI 192.168.206.121@tcp132 [8/256/0/180] [ 6786.497405] LNet: Added LNI 192.168.206.121@tcp133 [8/256/0/180] [ 6786.787375] LNet: Added LNI 192.168.206.121@tcp134 [8/256/0/180] [ 6787.100394] LNet: Added LNI 192.168.206.121@tcp135 [8/256/0/180] [ 6789.297388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6789.306593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.346074] Lustre: 463241:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [ 6790.407790] LNetError: 460706:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6790.435323] LNetError: Protocol error connecting to 192.168.206.121@tcp120 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6790.439948] LNetError: Skipped 1 previous similar message [ 6790.485343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6790.496348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6790.536452] Lustre: lustre-MDT0000: new disk, initializing [ 6790.559285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6790.565979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6790.815092] LNet: 5 local NIs in recovery (showing 5): 192.168.206.121@tcp127, 192.168.206.121@tcp129, 192.168.206.121@tcp120, 192.168.206.121@tcp124, 192.168.206.121@tcp128 [ 6791.455946] LNetError: 463276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6791.458611] LNetError: 463276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 6791.461574] LNetError: Protocol error connecting to 192.168.206.121@tcp110 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6791.463958] LNetError: Skipped 4 previous similar messages [ 6791.892626] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6793.890076] LNetError: 460707:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6793.893577] LNetError: 460707:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 6793.896585] LNetError: Protocol error connecting to 192.168.206.121@tcp127 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6793.900922] LNetError: Skipped 8 previous similar messages [ 6796.188339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.247113] Lustre: 463257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6796.281847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6796.284444] Lustre: Skipped 1 previous similar message [ 6796.327226] Lustre: lustre-MDT0001: new disk, initializing [ 6796.340528] Lustre: 464063:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [ 6796.343777] Lustre: 464063:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 6796.397414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6796.414230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6796.418143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6797.803519] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6797.984545] LNetError: 463268:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6797.987521] LNetError: 463268:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6797.990949] LNetError: Protocol error connecting to 192.168.206.121@tcp127 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6800.054180] Lustre: 464494:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [ 6800.057580] Lustre: 464494:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 6801.600139] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.402427] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6805.153809] LNetError: 463263:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6805.158334] LNetError: Protocol error connecting to 192.168.206.121@tcp127 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6805.595552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.695122] Lustre: lustre-OST0000: new disk, initializing [ 6805.696669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6805.699412] Lustre: 465236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6805.733085] Lustre: 465241:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [ 6805.735984] Lustre: 465241:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 6805.755964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.599209] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6811.194832] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.615076] Lustre: server umount lustre-OST0000 complete [ 6815.214175] Lustre: 466378:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [ 6815.216585] Lustre: 466378:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 6815.223866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.224880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.226907] Lustre: Skipped 1 previous similar message [ 6820.319959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.321780] Lustre: Skipped 2 previous similar messages [ 6821.011261] Lustre: server umount lustre-MDT0000 complete [ 6821.535994] LNetError: 463275:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.121 [ 6821.538334] LNetError: Protocol error connecting to 192.168.206.121@tcp127 at host 192.168.206.121:988: is it running a compatible version of Lustre? [ 6822.171078] LustreError: 463249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794465 with bad export cookie 3013556424995085038 [ 6822.173838] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.174116] LustreError: 463249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6822.285309] Lustre: server umount lustre-MDT0001 complete [ 6825.691101] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6826.763995] Key type lgssc unregistered [ 6826.921344] LNet: 467149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6826.923700] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6826.925917] LNet: Removed LNI 192.168.206.121@tcp [ 6827.935135] LNet: Removed LNI 192.168.206.121@tcp100 [ 6827.936667] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6827.939672] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6827.942433] LNet: Removed LNI 192.168.206.121@tcp101 [ 6828.959137] LNet: Removed LNI 192.168.206.121@tcp102 [ 6828.960619] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6828.962915] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6828.965230] LNet: Removed LNI 192.168.206.121@tcp103 [ 6829.983113] LNet: Removed LNI 192.168.206.121@tcp104 [ 6831.007135] LNet: Removed LNI 192.168.206.121@tcp105 [ 6831.008486] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6831.010432] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6831.012564] LNet: Removed LNI 192.168.206.121@tcp106 [ 6832.031132] LNet: Removed LNI 192.168.206.121@tcp107 [ 6833.055231] LNet: Removed LNI 192.168.206.121@tcp108 [ 6834.079121] LNet: Removed LNI 192.168.206.121@tcp109 [ 6835.103132] LNet: Removed LNI 192.168.206.121@tcp110 [ 6835.104890] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6835.107727] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6835.110777] LNet: Removed LNI 192.168.206.121@tcp111 [ 6836.127167] LNet: Removed LNI 192.168.206.121@tcp112 [ 6837.151115] LNet: Removed LNI 192.168.206.121@tcp113 [ 6838.175097] LNet: Removed LNI 192.168.206.121@tcp114 [ 6839.199134] LNet: Removed LNI 192.168.206.121@tcp115 [ 6840.223158] LNet: Removed LNI 192.168.206.121@tcp116 [ 6841.247094] LNet: Removed LNI 192.168.206.121@tcp117 [ 6842.271162] LNet: Removed LNI 192.168.206.121@tcp118 [ 6843.295151] LNet: Removed LNI 192.168.206.121@tcp119 [ 6843.296528] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6843.298753] LNetError: 467149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 6843.302069] LNet: Removed LNI 192.168.206.121@tcp120 [ 6844.319150] LNet: Removed LNI 192.168.206.121@tcp121 [ 6845.343140] LNet: Removed LNI 192.168.206.121@tcp122 [ 6846.367122] LNet: Removed LNI 192.168.206.121@tcp123 [ 6847.391104] LNet: Removed LNI 192.168.206.121@tcp124 [ 6848.415100] LNet: Removed LNI 192.168.206.121@tcp125 [ 6849.439125] LNet: Removed LNI 192.168.206.121@tcp126 [ 6849.440183] LNet: Removed LNI 192.168.206.121@tcp127 [ 6850.463116] LNet: Removed LNI 192.168.206.121@tcp128 [ 6851.487126] LNet: Removed LNI 192.168.206.121@tcp129 [ 6851.488276] LNet: Removed LNI 192.168.206.121@tcp130 [ 6851.489296] LNet: Removed LNI 192.168.206.121@tcp131 [ 6851.490343] LNet: Removed LNI 192.168.206.121@tcp132 [ 6851.491262] LNet: Removed LNI 192.168.206.121@tcp133 [ 6851.492148] LNet: Removed LNI 192.168.206.121@tcp134 [ 6851.499483] LNet: Removed LNI 192.168.206.121@tcp135 [ 6851.766528] Key type .llcrypt unregistered [ 6851.767550] Key type ._llcrypt unregistered [ 6859.285649] Key type ._llcrypt registered [ 6859.286500] Key type .llcrypt registered [ 6859.324809] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 6865.155786] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6865.495384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6865.533258] alg: No test for adler32 (adler32-zlib) [ 6866.400306] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6866.485773] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6868.063136] Key type lgssc registered [ 6868.438968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6879.394346] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:02:02 (1785794522) [ 6885.261463] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6886.237214] Key type lgssc unregistered [ 6886.355420] LNet: 471206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.357745] LNetError: 471206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6886.367473] LNet: Removed LNI 192.168.206.121@tcp [ 6886.679280] Key type .llcrypt unregistered [ 6886.680267] Key type ._llcrypt unregistered [ 6893.292927] Key type ._llcrypt registered [ 6893.294077] Key type .llcrypt registered [ 6893.331304] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6893.658907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6893.687537] alg: No test for adler32 (adler32-zlib) [ 6894.556223] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6894.659721] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6896.247210] Key type lgssc registered [ 6896.639635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6899.715378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6899.723699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.806652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6900.816985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6900.854802] Lustre: lustre-MDT0000: new disk, initializing [ 6900.874746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.880180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6902.241389] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6906.453369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.503299] Lustre: 472397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6906.534373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6906.536206] Lustre: Skipped 1 previous similar message [ 6906.572837] Lustre: lustre-MDT0001: new disk, initializing [ 6906.610150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.623450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6906.628984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6908.102503] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6911.936643] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6913.857969] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6916.213963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.322851] Lustre: lustre-OST0000: new disk, initializing [ 6916.324503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6916.327053] Lustre: 474366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6916.356536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.198861] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6919.096357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6919.099659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6919.132597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6921.847652] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6923.045244] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing lnet_if_list [ 6924.701710] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6924.713113] LNet: Added LNI 192.168.206.121@tcp10 [8/256/0/180] [ 6926.620912] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6926.630639] LNet: Added LNI 192.168.206.121@tcp11 [8/256/0/180] [ 6927.944283] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6927.954765] LNet: Added LNI 192.168.206.121@tcp12 [8/256/0/180] [ 6929.269249] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6929.279150] LNet: Added LNI 192.168.206.121@tcp13 [8/256/0/180] [ 6930.682623] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6930.695839] LNet: Added LNI 192.168.206.121@tcp14 [8/256/0/180] [ 6932.082317] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6932.093990] LNet: Added LNI 192.168.206.121@tcp15 [8/256/0/180] [ 6933.490106] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6933.501042] LNet: Added LNI 192.168.206.121@tcp16 [8/256/0/180] [ 6935.003850] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6935.015545] LNet: Added LNI 192.168.206.121@tcp17 [8/256/0/180] [ 6936.509359] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6936.519399] LNet: Added LNI 192.168.206.121@tcp18 [8/256/0/180] [ 6938.037057] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6938.048689] LNet: Added LNI 192.168.206.121@tcp19 [8/256/0/180] [ 6939.649082] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6939.658791] LNet: Added LNI 192.168.206.121@tcp20 [8/256/0/180] [ 6941.257811] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6941.270869] LNet: Added LNI 192.168.206.121@tcp21 [8/256/0/180] [ 6942.887054] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6942.897898] LNet: Added LNI 192.168.206.121@tcp22 [8/256/0/180] [ 6944.620240] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6944.632356] LNet: Added LNI 192.168.206.121@tcp23 [8/256/0/180] [ 6946.455769] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6946.467145] LNet: Added LNI 192.168.206.121@tcp24 [8/256/0/180] [ 6948.320547] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6948.330974] LNet: Added LNI 192.168.206.121@tcp25 [8/256/0/180] [ 6950.104222] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6950.115737] LNet: Added LNI 192.168.206.121@tcp26 [8/256/0/180] [ 6951.977017] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6951.990990] LNet: Added LNI 192.168.206.121@tcp27 [8/256/0/180] [ 6953.793431] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6953.803765] LNet: Added LNI 192.168.206.121@tcp28 [8/256/0/180] [ 6955.857215] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6955.869409] LNet: Added LNI 192.168.206.121@tcp29 [8/256/0/180] [ 6957.844153] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6957.856773] LNet: Added LNI 192.168.206.121@tcp30 [8/256/0/180] [ 6959.802189] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6959.813337] LNet: Added LNI 192.168.206.121@tcp31 [8/256/0/180] [ 6961.736858] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6961.747133] LNet: Added LNI 192.168.206.121@tcp32 [8/256/0/180] [ 6963.782383] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6963.793796] LNet: Added LNI 192.168.206.121@tcp33 [8/256/0/180] [ 6965.870590] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6965.881880] LNet: Added LNI 192.168.206.121@tcp34 [8/256/0/180] [ 6967.996813] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6968.007931] LNet: Added LNI 192.168.206.121@tcp35 [8/256/0/180] [ 6970.170616] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6970.181669] LNet: Added LNI 192.168.206.121@tcp36 [8/256/0/180] [ 6972.457578] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6972.466989] LNet: Added LNI 192.168.206.121@tcp37 [8/256/0/180] [ 6974.689579] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6974.701504] LNet: Added LNI 192.168.206.121@tcp38 [8/256/0/180] [ 6977.078275] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6977.091608] LNet: Added LNI 192.168.206.121@tcp39 [8/256/0/180] [ 6979.327594] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6979.338798] LNet: Added LNI 192.168.206.121@tcp40 [8/256/0/180] [ 6981.740817] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6981.751595] LNet: Added LNI 192.168.206.121@tcp41 [8/256/0/180] [ 6984.385100] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6984.397148] LNet: Added LNI 192.168.206.121@tcp42 [8/256/0/180] [ 6986.992377] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6987.004172] LNet: Added LNI 192.168.206.121@tcp43 [8/256/0/180] [ 6989.645447] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6989.658083] LNet: Added LNI 192.168.206.121@tcp44 [8/256/0/180] [ 6992.312504] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6992.322597] LNet: Added LNI 192.168.206.121@tcp45 [8/256/0/180] [ 6994.746164] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 6994.756662] LNet: Added LNI 192.168.206.121@tcp46 [8/256/0/180] [ 6997.312855] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 6997.324281] LNet: Added LNI 192.168.206.121@tcp47 [8/256/0/180] [ 7000.049910] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7000.064595] LNet: Added LNI 192.168.206.121@tcp48 [8/256/0/180] [ 7003.075786] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7003.088934] LNet: Added LNI 192.168.206.121@tcp49 [8/256/0/180] [ 7005.880352] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7005.895786] LNet: Added LNI 192.168.206.121@tcp50 [8/256/0/180] [ 7014.367761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7014.369965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.373824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7016.416260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7016.417144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.422290] Lustre: Skipped 1 previous similar message [ 7019.158278] Lustre: server umount lustre-MDT0000 complete [ 7020.302062] LustreError: 472389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794663 with bad export cookie 4976621435180482880 [ 7020.304572] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.304827] LustreError: 472389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7020.409920] Lustre: server umount lustre-MDT0001 complete [ 7027.207064] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7030.218109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.687418] LustreError: 471668:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000775ecd8e x1872541292513920/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7030.813378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7032.232661] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7035.091174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.219045] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.255686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7036.260861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7036.711326] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7037.280068] Lustre: 471672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794664/real 1785794664] req@000000007e8902dc x1872541292513536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785794680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7039.030631] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7040.799167] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7041.359129] Lustre: 471670:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794668/real 1785794668] req@000000007f62b707 x1872541292513664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785794684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7042.399122] Lustre: 471670:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794669/real 1785794669] req@000000005b43a934 x1872541292513792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785794685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7047.647868] Lustre: 471671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794673/real 1785794673] req@00000000dfcb6013 x1872541292514048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785794689 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7050.720899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.723687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.727904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7050.729702] Lustre: Skipped 2 previous similar messages [ 7053.783595] Lustre: server umount lustre-OST0000 complete [ 7055.840075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7055.843517] LustreError: Skipped 1 previous similar message [ 7055.845768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.850705] Lustre: Skipped 1 previous similar message [ 7055.852736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.854348] Lustre: Skipped 2 previous similar messages [ 7061.211434] Lustre: server umount lustre-MDT0000 complete [ 7061.984155] LustreError: 498553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.991236] LustreError: 498553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7062.520503] LustreError: 472389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794705 with bad export cookie 4976621435180484945 [ 7062.524408] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.526042] LustreError: 472389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7062.653100] Lustre: server umount lustre-MDT0001 complete [ 7066.128458] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7067.154333] Key type lgssc unregistered [ 7067.311509] LNet: 501327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7067.314967] LNetError: 501327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.317082] LNet: Removed LNI 192.168.206.121@tcp [ 7067.318449] LNet: Removed LNI 192.168.206.121@tcp10 [ 7067.319569] LNet: Removed LNI 192.168.206.121@tcp11 [ 7067.320602] LNet: Removed LNI 192.168.206.121@tcp12 [ 7067.321708] LNet: Removed LNI 192.168.206.121@tcp13 [ 7067.322844] LNet: Removed LNI 192.168.206.121@tcp14 [ 7067.323916] LNet: Removed LNI 192.168.206.121@tcp15 [ 7067.324865] LNet: Removed LNI 192.168.206.121@tcp16 [ 7067.325965] LNet: Removed LNI 192.168.206.121@tcp17 [ 7067.326988] LNet: Removed LNI 192.168.206.121@tcp18 [ 7067.327966] LNet: Removed LNI 192.168.206.121@tcp19 [ 7067.329095] LNet: Removed LNI 192.168.206.121@tcp20 [ 7067.330157] LNet: Removed LNI 192.168.206.121@tcp21 [ 7067.331214] LNet: Removed LNI 192.168.206.121@tcp22 [ 7067.332209] LNet: Removed LNI 192.168.206.121@tcp23 [ 7067.333250] LNet: Removed LNI 192.168.206.121@tcp24 [ 7067.334339] LNet: Removed LNI 192.168.206.121@tcp25 [ 7067.335480] LNet: Removed LNI 192.168.206.121@tcp26 [ 7067.336550] LNet: Removed LNI 192.168.206.121@tcp27 [ 7067.337700] LNet: Removed LNI 192.168.206.121@tcp28 [ 7067.338775] LNet: Removed LNI 192.168.206.121@tcp29 [ 7067.339806] LNet: Removed LNI 192.168.206.121@tcp30 [ 7067.341052] LNet: Removed LNI 192.168.206.121@tcp31 [ 7067.342257] LNet: Removed LNI 192.168.206.121@tcp32 [ 7067.343506] LNet: Removed LNI 192.168.206.121@tcp33 [ 7067.344801] LNet: Removed LNI 192.168.206.121@tcp34 [ 7067.345890] LNet: Removed LNI 192.168.206.121@tcp35 [ 7067.347189] LNet: Removed LNI 192.168.206.121@tcp36 [ 7067.348271] LNet: Removed LNI 192.168.206.121@tcp37 [ 7067.349518] LNet: Removed LNI 192.168.206.121@tcp38 [ 7067.350791] LNet: Removed LNI 192.168.206.121@tcp39 [ 7067.352122] LNet: Removed LNI 192.168.206.121@tcp40 [ 7067.353355] LNet: Removed LNI 192.168.206.121@tcp41 [ 7067.354342] LNet: Removed LNI 192.168.206.121@tcp42 [ 7067.355301] LNet: Removed LNI 192.168.206.121@tcp43 [ 7067.356304] LNet: Removed LNI 192.168.206.121@tcp44 [ 7067.357335] LNet: Removed LNI 192.168.206.121@tcp45 [ 7067.358313] LNet: Removed LNI 192.168.206.121@tcp46 [ 7067.359480] LNet: Removed LNI 192.168.206.121@tcp47 [ 7067.362942] LNet: Removed LNI 192.168.206.121@tcp48 [ 7067.363936] LNet: Removed LNI 192.168.206.121@tcp49 [ 7067.371370] LNet: Removed LNI 192.168.206.121@tcp50 [ 7067.651298] Key type .llcrypt unregistered [ 7067.652746] Key type ._llcrypt unregistered [ 7074.044571] Key type ._llcrypt registered [ 7074.045566] Key type .llcrypt registered [ 7074.084072] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7080.358578] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7080.826928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7080.841101] alg: No test for adler32 (adler32-zlib) [ 7081.697807] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7081.800159] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7083.383163] Key type lgssc registered [ 7083.769718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7095.751813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7095.757380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.846049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7096.857043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7096.900092] Lustre: lustre-MDT0000: new disk, initializing [ 7096.923307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.929398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7098.374243] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7102.309423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.347693] Lustre: 504496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7102.368870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7102.371832] Lustre: Skipped 1 previous similar message [ 7102.408598] Lustre: lustre-MDT0001: new disk, initializing [ 7102.440430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7102.451069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7102.454512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7103.901192] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7107.650491] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7109.565356] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7112.266625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.348242] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7112.377611] Lustre: lustre-OST0000: new disk, initializing [ 7112.379625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7112.382198] Lustre: 506509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7112.410974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.976301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7113.980990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7114.011586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7114.382279] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7118.172771] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7124.450574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.451099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.453163] LustreError: Skipped 1 previous similar message [ 7124.457123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7124.459507] Lustre: Skipped 1 previous similar message [ 7126.743670] Lustre: server umount lustre-OST0000 complete [ 7128.031780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7128.034115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.036877] Lustre: Skipped 1 previous similar message [ 7128.038314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7129.567949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7129.568703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.687968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.689436] Lustre: Skipped 2 previous similar messages [ 7139.808809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.810760] Lustre: Skipped 1 previous similar message [ 7142.367133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7142.438647] Lustre: server umount lustre-MDT0000 complete [ 7143.684195] LustreError: 504487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794786 with bad export cookie 2719846990528370677 [ 7143.687216] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.687917] LustreError: 504487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7143.801932] Lustre: server umount lustre-MDT0001 complete [ 7147.566566] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7148.658378] Key type lgssc unregistered [ 7148.791406] LNet: 508412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7148.793772] LNetError: 508412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7148.800327] LNet: Removed LNI 192.168.206.121@tcp [ 7149.139806] Key type .llcrypt unregistered [ 7149.140868] Key type ._llcrypt unregistered [ 7156.745046] Key type ._llcrypt registered [ 7156.746152] Key type .llcrypt registered [ 7156.783820] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7157.169854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7157.196595] alg: No test for adler32 (adler32-zlib) [ 7158.057321] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7158.154038] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7159.735147] Key type lgssc registered [ 7160.113941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7162.823437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7162.829153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.985275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.377060] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7168.268793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.395162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.813974] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7172.272972] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7174.147395] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7176.448583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.579473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.609825] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7180.852912] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.260324] loop0: detected capacity change from 0 to 204800000 [ 7195.368254] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7195.372323] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7195.375773] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.259428] loop0: detected capacity change from 0 to 204800000 [ 7197.370980] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.374870] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.378525] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7199.019880] loop0: detected capacity change from 0 to 204800000 [ 7200.819392] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.874905] Lustre: fs15246-OST0000: new disk, initializing [ 7200.877745] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7200.882209] Lustre: 513269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7200.903783] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.881750] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7206.826044] loop1: detected capacity change from 0 to 204800000 [ 7208.760970] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.799722] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7208.824401] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7208.869923] Lustre: fs15246-MDT0000: new disk, initializing [ 7208.933820] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.946042] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7208.992648] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7210.015844] Lustre: 514390:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000997955db x1872541569286400/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785794869 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7210.466176] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7215.072988] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7215.077358] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7215.083485] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7219.684526] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.689312] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.688420] Lustre: server umount fs15246-MDT0000 complete [ 7224.802739] LustreError: 511496:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.807065] LustreError: 511496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7226.631537] Lustre: server umount fs15246-OST0000 complete [ 7229.920849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.922084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.923292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.926916] Lustre: Skipped 1 previous similar message [ 7235.046757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7235.049122] Lustre: Skipped 2 previous similar messages [ 7240.162756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7240.165177] Lustre: Skipped 1 previous similar message [ 7244.255172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7244.309976] Lustre: server umount lustre-OST0000 complete [ 7250.402217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7250.402494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.403076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7250.409887] Lustre: Skipped 1 previous similar message [ 7251.804135] Lustre: server umount lustre-MDT0000 complete [ 7253.119610] LustreError: 509638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794896 with bad export cookie 1237782500146071358 [ 7253.123188] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.124518] LustreError: 509638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7253.246622] Lustre: server umount lustre-MDT0001 complete [ 7256.772326] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7257.688756] Key type lgssc unregistered [ 7257.810318] LNet: 517484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7257.812503] LNetError: 517484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7257.821311] LNet: Removed LNI 192.168.206.121@tcp [ 7258.119556] Key type .llcrypt unregistered [ 7258.120434] Key type ._llcrypt unregistered [ 7265.450470] Key type ._llcrypt registered [ 7265.451391] Key type .llcrypt registered [ 7265.495931] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7265.843586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7265.896244] alg: No test for adler32 (adler32-zlib) [ 7266.778715] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7266.885369] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7268.471159] Key type lgssc registered [ 7268.912835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.697065] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7284.898827] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7296.627854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7296.635598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.729248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7297.739654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7297.775773] Lustre: lustre-MDT0000: new disk, initializing [ 7297.796161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.802696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7299.148181] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7303.321566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.366988] Lustre: 521918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7303.390587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7303.392778] Lustre: Skipped 1 previous similar message [ 7303.432384] Lustre: lustre-MDT0001: new disk, initializing [ 7303.466280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7303.479049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7303.482108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7304.911594] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7308.683916] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7310.535867] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7312.833540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.934898] Lustre: lustre-OST0000: new disk, initializing [ 7312.937188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.940668] Lustre: 523887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7312.974405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.016444] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7315.833450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7315.837425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7315.868770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.917035] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7320.298932] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7320.369197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7321.586131] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7321.650504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7324.303468] Lustre: server umount lustre-OST0000 complete [ 7326.176581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.177491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.180320] Lustre: Skipped 1 previous similar message [ 7326.184877] Lustre: Skipped 1 previous similar message [ 7331.296067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.298547] Lustre: Skipped 1 previous similar message [ 7335.416980] Lustre: server umount lustre-MDT0000 complete [ 7336.653371] LustreError: 522714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794979 with bad export cookie 1912536447948818751 [ 7336.657476] LustreError: 522714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7336.657530] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.194868] Lustre: server umount lustre-MDT0001 complete [ 7354.064336] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7357.550509] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7367.885134] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7216 sec ========== 18:10:11 (1785795011) [ 7368.423908] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:10:11 (1785795011) === [ 7369.520306] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:10:12 (1785795012) === [ 7382.236758] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7383.290372] Key type lgssc unregistered [ 7383.412274] LNet: 530516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7383.414766] LNetError: 530516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7383.424274] LNet: Removed LNI 192.168.206.121@tcp [ 7383.730340] Key type .llcrypt unregistered [ 7383.731483] Key type ._llcrypt unregistered